[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:51:47 dist-test-slave-zj8q krb5kdc[26458](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:51:47 dist-test-slave-zj8q krb5kdc[26458](info): set up 2 sockets
Aug 06 02:51:47 dist-test-slave-zj8q krb5kdc[26458](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260806 02:51:48.408213 26446 mini_kdc.cc:121] Time spent starting KDC: real 1.186s user 0.003s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Aug 06 02:51:48 dist-test-slave-zj8q krb5kdc[26458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:51:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:51:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.211.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab.
Entry for principal kudu/127.25.211.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab.
I20260806 02:51:48.610746 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:43731
--webserver_interface=127.25.211.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:44209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.211.190:43731
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:51:49.019467 26483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:51:49.019795 26483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:51:49.019981 26483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:51:49.028438 26483 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:51:49.028569 26483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:51:49.028625 26483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:51:49.028721 26483 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:51:49.028782 26483 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:51:49.040112 26483 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:44209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.211.190:43731
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:43731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.211.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:51:49.041708 26483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:51:49.043300 26483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:51:49.060129 26491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:49.060109 26493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:49.060109 26495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:49.060984 26483 server_base.cc:1034] running on GCE node
I20260806 02:51:49.061929 26483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:51:49.063761 26483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:51:49.065135 26483 hybrid_clock.cc:648] HybridClock initialized: now 1785984709064950 us; error 65 us; skew 500 ppm
I20260806 02:51:49.065636 26483 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:51:49 dist-test-slave-zj8q krb5kdc[26458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984709, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:51:49.074218 26483 init.cc:377] Logged in from keytab as kudu/127.25.211.190@KRBTEST.COM (short username kudu)
I20260806 02:51:49.076087 26483 webserver.cc:460] Webserver started at http://127.25.211.190:35641/ using document root <none> and password file <none>
I20260806 02:51:49.077046 26483 fs_manager.cc:359] Metadata directory not provided
I20260806 02:51:49.077277 26483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:51:49.077626 26483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:51:49.081416 26483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/master-0/data/instance:
uuid: "45a55ec6402a41b4a426d1b5e1784264"
format_stamp: "Formatted at 2026-08-06 02:51:49 on dist-test-slave-zj8q"
server_key: "cb9de2fb92a314f0ba2c38c2d557c20b"
server_key_iv: "eb3532bef92acceb6f73ae262de6833f"
server_key_version: "encryptionkey@0"
I20260806 02:51:49.082302 26483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/master-0/wal/instance:
uuid: "45a55ec6402a41b4a426d1b5e1784264"
format_stamp: "Formatted at 2026-08-06 02:51:49 on dist-test-slave-zj8q"
server_key: "cb9de2fb92a314f0ba2c38c2d557c20b"
server_key_iv: "eb3532bef92acceb6f73ae262de6833f"
server_key_version: "encryptionkey@0"
I20260806 02:51:49.088670 26483 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 02:51:49.092880 26511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:49.094399 26483 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 02:51:49.094588 26483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/master-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/master-0/wal
uuid: "45a55ec6402a41b4a426d1b5e1784264"
format_stamp: "Formatted at 2026-08-06 02:51:49 on dist-test-slave-zj8q"
server_key: "cb9de2fb92a314f0ba2c38c2d557c20b"
server_key_iv: "eb3532bef92acceb6f73ae262de6833f"
server_key_version: "encryptionkey@0"
I20260806 02:51:49.094802 26483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:51:49.119753 26483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:51:49.124590 26483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:51:49.124907 26483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:51:49.150858 26483 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.190:43731
I20260806 02:51:49.150878 26593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.190:43731 every 8 connection(s)
I20260806 02:51:49.152340 26483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
I20260806 02:51:49.156683 26594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:51:49.163137 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 26483
I20260806 02:51:49.163437 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/master-0/wal/instance
I20260806 02:51:49.165537 26446 external_mini_cluster.cc:1428] Setting key e1b7c8d1b8893eda900612e8ff7de821
I20260806 02:51:49.167771 26594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45a55ec6402a41b4a426d1b5e1784264: Bootstrap starting.
I20260806 02:51:49.173040 26594 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45a55ec6402a41b4a426d1b5e1784264: Neither blocks nor log segments found. Creating new log.
I20260806 02:51:49.174333 26594 log.cc:824] T 00000000000000000000000000000000 P 45a55ec6402a41b4a426d1b5e1784264: Log is configured to *not* fsync() on all Append() calls
I20260806 02:51:49.177192 26594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45a55ec6402a41b4a426d1b5e1784264: No bootstrap required, opened a new log
I20260806 02:51:49.187083 26594 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45a55ec6402a41b4a426d1b5e1784264 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45a55ec6402a41b4a426d1b5e1784264" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 43731 } }
I20260806 02:51:49.187521 26594 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45a55ec6402a41b4a426d1b5e1784264 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:51:49.187621 26594 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45a55ec6402a41b4a426d1b5e1784264 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45a55ec6402a41b4a426d1b5e1784264, State: Initialized, Role: FOLLOWER
I20260806 02:51:49.188198 26594 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45a55ec6402a41b4a426d1b5e1784264 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45a55ec6402a41b4a426d1b5e1784264" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 43731 } }
I20260806 02:51:49.188441 26594 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45a55ec6402a41b4a426d1b5e1784264 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:51:49.188573 26594 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45a55ec6402a41b4a426d1b5e1784264 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:51:49.188740 26594 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45a55ec6402a41b4a426d1b5e1784264 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:51:49.191182 26594 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45a55ec6402a41b4a426d1b5e1784264 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45a55ec6402a41b4a426d1b5e1784264" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 43731 } }
I20260806 02:51:49.191787 26594 leader_election.cc:302] T 00000000000000000000000000000000 P 45a55ec6402a41b4a426d1b5e1784264 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45a55ec6402a41b4a426d1b5e1784264; no voters:
I20260806 02:51:49.192497 26594 leader_election.cc:288] T 00000000000000000000000000000000 P 45a55ec6402a41b4a426d1b5e1784264 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:51:49.192608 26601 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45a55ec6402a41b4a426d1b5e1784264 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:51:49.193674 26601 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45a55ec6402a41b4a426d1b5e1784264 [term 1 LEADER]: Becoming Leader. State: Replica: 45a55ec6402a41b4a426d1b5e1784264, State: Running, Role: LEADER
I20260806 02:51:49.194399 26601 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45a55ec6402a41b4a426d1b5e1784264 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45a55ec6402a41b4a426d1b5e1784264" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 43731 } }
I20260806 02:51:49.194916 26594 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45a55ec6402a41b4a426d1b5e1784264 [sys.catalog]: configured and running, proceeding with master startup.
Aug 06 02:51:49 dist-test-slave-zj8q krb5kdc[26458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984708, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:51:49.197613 26603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45a55ec6402a41b4a426d1b5e1784264 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45a55ec6402a41b4a426d1b5e1784264. Latest consensus state: current_term: 1 leader_uuid: "45a55ec6402a41b4a426d1b5e1784264" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45a55ec6402a41b4a426d1b5e1784264" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 43731 } } }
I20260806 02:51:49.197983 26603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45a55ec6402a41b4a426d1b5e1784264 [sys.catalog]: This master's current role is: LEADER
I20260806 02:51:49.198312 26602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45a55ec6402a41b4a426d1b5e1784264 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45a55ec6402a41b4a426d1b5e1784264" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45a55ec6402a41b4a426d1b5e1784264" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 43731 } } }
I20260806 02:51:49.198683 26602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45a55ec6402a41b4a426d1b5e1784264 [sys.catalog]: This master's current role is: LEADER
I20260806 02:51:49.208958 26609 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:51:49.211910 26598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:49.169598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41466 (local address 127.25.211.190:43731)
0806 02:51:49.170723 (+ 1125us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:49.170738 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:49.172371 (+ 1633us) server_negotiation.cc:407] Connection header received
0806 02:51:49.182576 (+ 10205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:49.182603 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:49.183064 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:49.183487 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:49.186882 (+ 3395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:49.188470 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:49.190293 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:49.191033 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:49.191219 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:49.198964 (+ 7745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:49.198983 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:49.199007 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:49.199252 (+ 245us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:49.204197 (+ 4945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:49.205501 (+ 1304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:49.205537 (+ 36us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:49.205546 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:49.205818 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:49.206595 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:49.206601 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:49.206605 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:49.207579 (+ 974us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:49.207912 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:49.209772 (+ 1860us) server_negotiation.cc:299] Negotiation successful
0806 02:51:49.210271 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":850,"thread_start_us":620,"threads_started":1}
I20260806 02:51:49.215592 26609 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:51:49.225749 26609 catalog_manager.cc:1347] Generated new cluster ID: fbdf5bf9dc1c479084ff32b585704b37
I20260806 02:51:49.225888 26609 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:51:49.239212 26609 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:51:49.240808 26609 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:51:49.253721 26609 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45a55ec6402a41b4a426d1b5e1784264: Generated new TSK 0
I20260806 02:51:49.255026 26609 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.211.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab.
Entry for principal kudu/127.25.211.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab.
I20260806 02:51:49.314929 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.129:0
--local_ip_for_outbound_sockets=127.25.211.129
--webserver_interface=127.25.211.129
--webserver_port=0
--tserver_master_addrs=127.25.211.190:43731
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab
--principal=kudu/127.25.211.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:44209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:51:49.732749 26636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:51:49.733080 26636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:51:49.733233 26636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:51:49.741122 26636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:51:49.741288 26636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:51:49.741511 26636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.129
I20260806 02:51:49.752787 26636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:44209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab
--principal=kudu/127.25.211.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.211.129
--webserver_port=0
--tserver_master_addrs=127.25.211.190:43731
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.129
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:51:49.754521 26636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:51:49.756053 26636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:51:49.770483 26643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:49.770457 26646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:49.770510 26644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:49.771925 26636 server_base.cc:1034] running on GCE node
I20260806 02:51:49.772675 26636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:51:49.773808 26636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:51:49.775020 26636 hybrid_clock.cc:648] HybridClock initialized: now 1785984709774964 us; error 50 us; skew 500 ppm
I20260806 02:51:49.775372 26636 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:51:49 dist-test-slave-zj8q krb5kdc[26458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984709, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:51:49.779374 26636 init.cc:377] Logged in from keytab as kudu/127.25.211.129@KRBTEST.COM (short username kudu)
I20260806 02:51:49.781287 26636 webserver.cc:460] Webserver started at http://127.25.211.129:38683/ using document root <none> and password file <none>
I20260806 02:51:49.782066 26636 fs_manager.cc:359] Metadata directory not provided
I20260806 02:51:49.782239 26636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:51:49.782588 26636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:51:49.785857 26636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/ts-0/data/instance:
uuid: "e654cac89a64437c9abc3b1c7509367d"
format_stamp: "Formatted at 2026-08-06 02:51:49 on dist-test-slave-zj8q"
server_key: "4fd08829e18b59b8a923e11f39172478"
server_key_iv: "46ee9217cdc32397f658514bf9434518"
server_key_version: "encryptionkey@0"
I20260806 02:51:49.786768 26636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/ts-0/wal/instance:
uuid: "e654cac89a64437c9abc3b1c7509367d"
format_stamp: "Formatted at 2026-08-06 02:51:49 on dist-test-slave-zj8q"
server_key: "4fd08829e18b59b8a923e11f39172478"
server_key_iv: "46ee9217cdc32397f658514bf9434518"
server_key_version: "encryptionkey@0"
I20260806 02:51:49.793288 26636 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260806 02:51:49.797716 26655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:49.799400 26636 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:51:49.799599 26636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/ts-0/wal
uuid: "e654cac89a64437c9abc3b1c7509367d"
format_stamp: "Formatted at 2026-08-06 02:51:49 on dist-test-slave-zj8q"
server_key: "4fd08829e18b59b8a923e11f39172478"
server_key_iv: "46ee9217cdc32397f658514bf9434518"
server_key_version: "encryptionkey@0"
I20260806 02:51:49.799824 26636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:51:49.813611 26636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:51:49.818521 26636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:51:49.818817 26636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:51:49.819945 26636 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:51:49.821394 26636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:51:49.821478 26636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:49.821653 26636 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:51:49.821786 26636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:49.865921 26636 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.129:40689
I20260806 02:51:49.865988 26829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.129:40689 every 8 connection(s)
I20260806 02:51:49.867759 26636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
I20260806 02:51:49.869098 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 26636
I20260806 02:51:49.869357 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/ts-0/wal/instance
I20260806 02:51:49.870018 26446 external_mini_cluster.cc:1428] Setting key 65faa203cba173928309cb35133d0e52
Aug 06 02:51:49 dist-test-slave-zj8q krb5kdc[26458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984709, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.129@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.211.130@KRBTEST.COM; defaulting to no policy
I20260806 02:51:49.893961 26836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:49.870883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:38347 (local address 127.25.211.190:43731)
0806 02:51:49.871448 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:49.871453 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:49.873283 (+ 1830us) server_negotiation.cc:407] Connection header received
0806 02:51:49.876510 (+ 3227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:49.876515 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:49.876588 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:49.876701 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:49.879095 (+ 2394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:49.879902 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:49.881959 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:49.882603 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:49.882735 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:49.887843 (+ 5108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:49.887850 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:49.887853 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:49.887894 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:49.890501 (+ 2607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:49.891391 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:49.891396 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:49.891398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:49.891548 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:49.892036 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:49.892040 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:49.892042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:49.892346 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:49.892490 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:49.893333 (+ 843us) server_negotiation.cc:299] Negotiation successful
0806 02:51:49.893586 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":256,"threads_started":1}
I20260806 02:51:49.895201 26837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:49.871477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:43731 (local address 127.25.211.129:38347)
0806 02:51:49.872930 (+ 1453us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:49.873026 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:49.876127 (+ 3101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:49.877068 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:49.877084 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:49.877893 (+ 809us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:49.878847 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:49.878877 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:49.880098 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:49.880103 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:49.881770 (+ 1667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:49.881791 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:49.882994 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:49.883003 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:49.883320 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:49.884078 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:49.884108 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:49.887504 (+ 3396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:51:49.890695 (+ 3191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:49.890705 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:49.890722 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:49.891241 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:49.891680 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:49.891684 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:49.891687 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:49.891905 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:49.892486 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:51:49.892496 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:51:49.892914 (+ 418us) client_negotiation.cc:769] Sending connection context
0806 02:51:49.893176 (+ 262us) client_negotiation.cc:240] Negotiation successful
0806 02:51:49.893805 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1068,"thread_start_us":426,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.130@KRBTEST.COM" created.
I20260806 02:51:49.897193 26831 heartbeater.cc:347] Connected to a master server at 127.25.211.190:43731
I20260806 02:51:49.897568 26831 heartbeater.cc:464] Registering TS with master...
I20260806 02:51:49.898561 26831 heartbeater.cc:511] Master 127.25.211.190:43731 requested a full tablet report, sending...
I20260806 02:51:49.901628 26536 ts_manager.cc:194] Registered new tserver with Master: e654cac89a64437c9abc3b1c7509367d (127.25.211.129:40689)
I20260806 02:51:49.903463 26536 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.211.129@KRBTEST.COM'} at 127.25.211.129:38347
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab.
Entry for principal kudu/127.25.211.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab.
I20260806 02:51:49.921439 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.130:0
--local_ip_for_outbound_sockets=127.25.211.130
--webserver_interface=127.25.211.130
--webserver_port=0
--tserver_master_addrs=127.25.211.190:43731
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab
--principal=kudu/127.25.211.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:44209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:51:50.337843 26844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:51:50.338196 26844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:51:50.338308 26844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:51:50.346611 26844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:51:50.346791 26844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:51:50.346937 26844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.130
I20260806 02:51:50.359104 26844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:44209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab
--principal=kudu/127.25.211.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.211.130
--webserver_port=0
--tserver_master_addrs=127.25.211.190:43731
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.130
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:51:50.360541 26844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:51:50.362207 26844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:51:50.377700 26858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:50.377710 26856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:50.378345 26852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:50.379091 26844 server_base.cc:1034] running on GCE node
I20260806 02:51:50.379829 26844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:51:50.380848 26844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:51:50.382046 26844 hybrid_clock.cc:648] HybridClock initialized: now 1785984710381978 us; error 38 us; skew 500 ppm
I20260806 02:51:50.382462 26844 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:51:50 dist-test-slave-zj8q krb5kdc[26458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984710, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:51:50.386315 26844 init.cc:377] Logged in from keytab as kudu/127.25.211.130@KRBTEST.COM (short username kudu)
I20260806 02:51:50.388113 26844 webserver.cc:460] Webserver started at http://127.25.211.130:34453/ using document root <none> and password file <none>
I20260806 02:51:50.388998 26844 fs_manager.cc:359] Metadata directory not provided
I20260806 02:51:50.389132 26844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:51:50.389456 26844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:51:50.392676 26844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/ts-1/data/instance:
uuid: "3ae839dae638464d865b95641043024b"
format_stamp: "Formatted at 2026-08-06 02:51:50 on dist-test-slave-zj8q"
server_key: "36571f5b2553187363ca823e5ba6ee85"
server_key_iv: "238ca791768cf13333f58381e9069d27"
server_key_version: "encryptionkey@0"
I20260806 02:51:50.393632 26844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/ts-1/wal/instance:
uuid: "3ae839dae638464d865b95641043024b"
format_stamp: "Formatted at 2026-08-06 02:51:50 on dist-test-slave-zj8q"
server_key: "36571f5b2553187363ca823e5ba6ee85"
server_key_iv: "238ca791768cf13333f58381e9069d27"
server_key_version: "encryptionkey@0"
I20260806 02:51:50.399415 26844 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260806 02:51:50.403182 26866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:50.404613 26844 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:51:50.404785 26844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/ts-1/wal
uuid: "3ae839dae638464d865b95641043024b"
format_stamp: "Formatted at 2026-08-06 02:51:50 on dist-test-slave-zj8q"
server_key: "36571f5b2553187363ca823e5ba6ee85"
server_key_iv: "238ca791768cf13333f58381e9069d27"
server_key_version: "encryptionkey@0"
I20260806 02:51:50.405018 26844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:51:50.449452 26844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:51:50.454073 26844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:51:50.454411 26844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:51:50.455677 26844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:51:50.457472 26844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:51:50.457551 26844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:50.457676 26844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:51:50.457726 26844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:50.499631 26844 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.130:32911
I20260806 02:51:50.499694 27043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.130:32911 every 8 connection(s)
I20260806 02:51:50.501072 26844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
I20260806 02:51:50.503391 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 26844
I20260806 02:51:50.503610 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/ts-1/wal/instance
I20260806 02:51:50.504242 26446 external_mini_cluster.cc:1428] Setting key 1c7d35710f79325949e0a814718cc4af
Aug 06 02:51:50 dist-test-slave-zj8q krb5kdc[26458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984710, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.130@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.211.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.131@KRBTEST.COM" created.
I20260806 02:51:50.528724 27046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:50.504291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:54555 (local address 127.25.211.190:43731)
0806 02:51:50.510669 (+ 6378us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:50.510674 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:50.510695 (+ 21us) server_negotiation.cc:407] Connection header received
0806 02:51:50.510754 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:50.510758 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:50.510825 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:50.510926 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:50.513756 (+ 2830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:50.514798 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:50.516546 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:50.517189 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:50.517325 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:50.521540 (+ 4215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:50.521548 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:50.521553 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:50.521603 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:50.525502 (+ 3899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:50.526449 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:50.526455 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:50.526459 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:50.526593 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:50.526995 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:50.526998 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:50.527000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:50.527299 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:50.527416 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:50.528131 (+ 715us) server_negotiation.cc:299] Negotiation successful
0806 02:51:50.528353 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6260,"thread_start_us":6137,"threads_started":1}
I20260806 02:51:50.529769 27047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:50.505015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:43731 (local address 127.25.211.130:54555)
0806 02:51:50.506237 (+ 1222us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:50.506294 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:50.508717 (+ 2423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:50.511365 (+ 2648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:50.511379 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:50.512355 (+ 976us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:50.513527 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:50.513563 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:50.515001 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:50.515005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:50.516369 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:50.516384 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:50.517432 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:50.517437 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:50.517744 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:50.518527 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:50.518559 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:50.521225 (+ 2666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:51:50.525713 (+ 4488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:50.525723 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:50.525741 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:50.526309 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:50.526693 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:50.526696 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:50.526698 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:50.526901 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:50.527401 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:51:50.527407 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:51:50.527779 (+ 372us) client_negotiation.cc:769] Sending connection context
0806 02:51:50.528014 (+ 235us) client_negotiation.cc:240] Negotiation successful
0806 02:51:50.528605 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":833,"thread_start_us":547,"threads_started":1}
I20260806 02:51:50.531312 27044 heartbeater.cc:347] Connected to a master server at 127.25.211.190:43731
I20260806 02:51:50.531662 27044 heartbeater.cc:464] Registering TS with master...
I20260806 02:51:50.532568 27044 heartbeater.cc:511] Master 127.25.211.190:43731 requested a full tablet report, sending...
I20260806 02:51:50.534260 26536 ts_manager.cc:194] Registered new tserver with Master: 3ae839dae638464d865b95641043024b (127.25.211.130:32911)
I20260806 02:51:50.535379 26536 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.211.130@KRBTEST.COM'} at 127.25.211.130:54555
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab.
Entry for principal kudu/127.25.211.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab.
I20260806 02:51:50.553859 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.131:0
--local_ip_for_outbound_sockets=127.25.211.131
--webserver_interface=127.25.211.131
--webserver_port=0
--tserver_master_addrs=127.25.211.190:43731
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab
--principal=kudu/127.25.211.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:44209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:51:50.906956 26831 heartbeater.cc:503] Master 127.25.211.190:43731 was elected leader, sending a full tablet report...
W20260806 02:51:50.954233 27052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:51:50.954566 27052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:51:50.954704 27052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:51:50.962837 27052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:51:50.962965 27052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:51:50.963121 27052 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.131
I20260806 02:51:50.974504 27052 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:44209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab
--principal=kudu/127.25.211.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.211.131
--webserver_port=0
--tserver_master_addrs=127.25.211.190:43731
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.131
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:51:50.975896 27052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:51:50.977396 27052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:51:50.992609 27061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:50.992931 27065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:50.992998 27063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:50.994138 27052 server_base.cc:1034] running on GCE node
I20260806 02:51:50.995064 27052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:51:50.996141 27052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:51:50.997357 27052 hybrid_clock.cc:648] HybridClock initialized: now 1785984710997305 us; error 43 us; skew 500 ppm
I20260806 02:51:50.997802 27052 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:51:50 dist-test-slave-zj8q krb5kdc[26458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984710, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:51:51.001627 27052 init.cc:377] Logged in from keytab as kudu/127.25.211.131@KRBTEST.COM (short username kudu)
I20260806 02:51:51.003213 27052 webserver.cc:460] Webserver started at http://127.25.211.131:34459/ using document root <none> and password file <none>
I20260806 02:51:51.004021 27052 fs_manager.cc:359] Metadata directory not provided
I20260806 02:51:51.004150 27052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:51:51.004462 27052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:51:51.007913 27052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/ts-2/data/instance:
uuid: "ee83b8293eb449f7b5fd98453ff6438c"
format_stamp: "Formatted at 2026-08-06 02:51:50 on dist-test-slave-zj8q"
server_key: "2c408a89d67ce62fe50211479fe94d29"
server_key_iv: "eb0988ca540d24ab17646d075903d97a"
server_key_version: "encryptionkey@0"
I20260806 02:51:51.008726 27052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/ts-2/wal/instance:
uuid: "ee83b8293eb449f7b5fd98453ff6438c"
format_stamp: "Formatted at 2026-08-06 02:51:50 on dist-test-slave-zj8q"
server_key: "2c408a89d67ce62fe50211479fe94d29"
server_key_iv: "eb0988ca540d24ab17646d075903d97a"
server_key_version: "encryptionkey@0"
I20260806 02:51:51.014371 27052 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260806 02:51:51.018500 27074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:51.020149 27052 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:51:51.020319 27052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/ts-2/wal
uuid: "ee83b8293eb449f7b5fd98453ff6438c"
format_stamp: "Formatted at 2026-08-06 02:51:50 on dist-test-slave-zj8q"
server_key: "2c408a89d67ce62fe50211479fe94d29"
server_key_iv: "eb0988ca540d24ab17646d075903d97a"
server_key_version: "encryptionkey@0"
I20260806 02:51:51.020519 27052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:51:51.057629 27052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:51:51.062383 27052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:51:51.062704 27052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:51:51.063678 27052 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:51:51.065135 27052 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:51:51.065209 27052 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:51.065307 27052 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:51:51.065387 27052 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:51.108383 27052 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.131:32845
I20260806 02:51:51.108383 27245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.131:32845 every 8 connection(s)
I20260806 02:51:51.110136 27052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
I20260806 02:51:51.115679 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 27052
I20260806 02:51:51.115871 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1785984707115702-26446-0/minicluster-data/ts-2/wal/instance
I20260806 02:51:51.116310 26446 external_mini_cluster.cc:1428] Setting key 066aa0a3fc56cc05cf283b6db5c36703
Aug 06 02:51:51 dist-test-slave-zj8q krb5kdc[26458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984710, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.131@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:51:51.134977 27251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:51.113242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:54287 (local address 127.25.211.190:43731)
0806 02:51:51.113879 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:51.113884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:51.114850 (+ 966us) server_negotiation.cc:407] Connection header received
0806 02:51:51.117276 (+ 2426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:51.117283 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:51.117363 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:51.117453 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:51.119521 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:51.120811 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:51.122980 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:51.123400 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:51.123511 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:51.127563 (+ 4052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:51.127570 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:51.127573 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:51.127612 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:51.130728 (+ 3116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:51.131884 (+ 1156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:51.131889 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:51.131891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:51.132035 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:51.132683 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:51.132687 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:51.132689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:51.133139 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:51.133297 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:51.134077 (+ 780us) server_negotiation.cc:299] Negotiation successful
0806 02:51:51.134367 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":364,"threads_started":1}
I20260806 02:51:51.135860 27252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:51.113877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:43731 (local address 127.25.211.131:54287)
0806 02:51:51.114684 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:51.114725 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:51.116920 (+ 2195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:51.117804 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:51.117817 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:51.118550 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:51.119330 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:51.119352 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:51.121003 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:51.121008 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:51.122806 (+ 1798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:51.122821 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:51.123645 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:51.123651 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:51.123880 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:51.124788 (+ 908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:51.124826 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:51.127299 (+ 2473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:51:51.130928 (+ 3629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:51.130939 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:51.130956 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:51.131741 (+ 785us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:51.132185 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:51.132188 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:51.132191 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:51.132534 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:51.133274 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:51:51.133283 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:51:51.133691 (+ 408us) client_negotiation.cc:769] Sending connection context
0806 02:51:51.133950 (+ 259us) client_negotiation.cc:240] Negotiation successful
0806 02:51:51.134595 (+ 645us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":359,"threads_started":1}
I20260806 02:51:51.137612 27246 heartbeater.cc:347] Connected to a master server at 127.25.211.190:43731
I20260806 02:51:51.137989 27246 heartbeater.cc:464] Registering TS with master...
I20260806 02:51:51.138857 27246 heartbeater.cc:511] Master 127.25.211.190:43731 requested a full tablet report, sending...
I20260806 02:51:51.140413 26536 ts_manager.cc:194] Registered new tserver with Master: ee83b8293eb449f7b5fd98453ff6438c (127.25.211.131:32845)
I20260806 02:51:51.141220 26536 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.211.131@KRBTEST.COM'} at 127.25.211.131:54287
I20260806 02:51:51.145686 26446 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 02:51:51 dist-test-slave-zj8q krb5kdc[26458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984711, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 02:51:51 dist-test-slave-zj8q krb5kdc[26458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984711, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.25.211.129@KRBTEST.COM
I20260806 02:51:51.199700 27260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:51.181394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44896 (local address 127.25.211.129:40689)
0806 02:51:51.182386 (+ 992us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:51.182395 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:51.182425 (+ 30us) server_negotiation.cc:407] Connection header received
0806 02:51:51.182495 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:51.182503 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:51.182751 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:51.182899 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:51.184497 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:51.186294 (+ 1797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:51.187848 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:51.188505 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:51.188640 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:51.193202 (+ 4562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:51.193220 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:51.193240 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:51.193291 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:51.196737 (+ 3446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:51.197554 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:51.197562 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:51.197565 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:51.197722 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:51.198089 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:51.198092 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:51.198094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:51.198530 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:51.198772 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:51.199072 (+ 300us) server_negotiation.cc:299] Negotiation successful
0806 02:51:51.199310 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":775,"thread_start_us":561,"threads_started":1}
W20260806 02:51:51.201305 26698 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44896
Aug 06 02:51:51 dist-test-slave-zj8q krb5kdc[26458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 02:51:51 dist-test-slave-zj8q krb5kdc[26458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.129@KRBTEST.COM
I20260806 02:51:51.252703 27260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:51.237727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44908 (local address 127.25.211.129:40689)
0806 02:51:51.237967 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:51.237974 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:51.238065 (+ 91us) server_negotiation.cc:407] Connection header received
0806 02:51:51.238314 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:51.238318 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:51.238389 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:51.238482 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:51.240018 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:51.240721 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:51.242127 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:51.242636 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:51.242757 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:51.247307 (+ 4550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:51.247313 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:51.247317 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:51.247359 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:51.249918 (+ 2559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:51.250750 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:51.250756 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:51.250758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:51.250909 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:51.251296 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:51.251299 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:51.251301 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:51.251571 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:51.251808 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:51.252144 (+ 336us) server_negotiation.cc:299] Negotiation successful
0806 02:51:51.252379 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260806 02:51:51.254858 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 26636
I20260806 02:51:51.270067 27260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:51.255881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44912 (local address 127.25.211.129:40689)
0806 02:51:51.256036 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:51.256040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:51.256345 (+ 305us) server_negotiation.cc:407] Connection header received
0806 02:51:51.256494 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:51.256499 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:51.256576 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:51.256660 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:51.257955 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:51.258846 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:51.260574 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:51.261019 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:51.261135 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:51.263691 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:51.263704 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:51.263707 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:51.263746 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:51.267215 (+ 3469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:51.268034 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:51.268039 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:51.268041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:51.268210 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:51.268581 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:51.268584 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:51.268586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:51.268893 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:51.269086 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:51.269425 (+ 339us) server_negotiation.cc:299] Negotiation successful
0806 02:51:51.269612 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 02:51:51.270884 26821 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:51:51.355360 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 26636
I20260806 02:51:51.378759 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 26844
Aug 06 02:51:51 dist-test-slave-zj8q krb5kdc[26458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.130@KRBTEST.COM
I20260806 02:51:51.395630 27268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:51.379756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43568 (local address 127.25.211.130:32911)
0806 02:51:51.380322 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:51.380330 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:51.380357 (+ 27us) server_negotiation.cc:407] Connection header received
0806 02:51:51.380419 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:51.380426 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:51.380652 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:51.380787 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:51.382094 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:51.383244 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:51.384608 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:51.385195 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:51.385307 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:51.389480 (+ 4173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:51.389494 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:51.389511 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:51.389558 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:51.392423 (+ 2865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:51.393140 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:51.393149 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:51.393152 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:51.393304 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:51.393762 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:51.393766 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:51.393769 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:51.394230 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:51.394643 (+ 413us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:51.394947 (+ 304us) server_negotiation.cc:299] Negotiation successful
0806 02:51:51.395234 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":293,"threads_started":1}
I20260806 02:51:51.396955 27036 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:51:51.479907 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 26844
I20260806 02:51:51.502040 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 27052
Aug 06 02:51:51 dist-test-slave-zj8q krb5kdc[26458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984711, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.131@KRBTEST.COM
I20260806 02:51:51.518640 27270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:51.503168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51464 (local address 127.25.211.131:32845)
0806 02:51:51.503854 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:51.503861 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:51.503890 (+ 29us) server_negotiation.cc:407] Connection header received
0806 02:51:51.503953 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:51.503959 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:51.504164 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:51.504319 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:51.505647 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:51.506865 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:51.508176 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:51.508755 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:51.508869 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:51.513008 (+ 4139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:51.513019 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:51.513036 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:51.513074 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:51.515810 (+ 2736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:51.516468 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:51.516475 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:51.516477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:51.516608 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:51.516967 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:51.516971 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:51.516973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:51.517364 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:51.517696 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:51.517972 (+ 276us) server_negotiation.cc:299] Negotiation successful
0806 02:51:51.518281 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":375,"threads_started":1}
I20260806 02:51:51.520063 27238 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:51:51.607100 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 27052
I20260806 02:51:51.629453 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 26483
I20260806 02:51:51.630726 26586 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:51:51.735515 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 26483
2026-08-06T02:51:51Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4572 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:51:51 dist-test-slave-zj8q krb5kdc[27278](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:51:51 dist-test-slave-zj8q krb5kdc[27278](info): set up 2 sockets
Aug 06 02:51:51 dist-test-slave-zj8q krb5kdc[27278](info): commencing operation
krb5kdc: starting...
W20260806 02:51:52.931411 26446 mini_kdc.cc:121] Time spent starting KDC: real 1.149s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Aug 06 02:51:53 dist-test-slave-zj8q krb5kdc[27278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984713, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:51:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:51:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.211.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab.
Entry for principal kudu/127.25.211.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab.
I20260806 02:51:53.150647 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:46793
--webserver_interface=127.25.211.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:42205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.211.190:46793
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:51:53.493916 27292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:51:53.494280 27292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:51:53.494418 27292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:51:53.502033 27292 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:51:53.502229 27292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:51:53.502333 27292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:51:53.502421 27292 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:51:53.502483 27292 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:51:53.512837 27292 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:42205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.211.190:46793
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:46793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.211.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:51:53.514249 27292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:51:53.515707 27292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:51:53.527868 27299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:53.527904 27300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:53.527942 27303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:53.528637 27292 server_base.cc:1034] running on GCE node
I20260806 02:51:53.529477 27292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:51:53.531119 27292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:51:53.532330 27292 hybrid_clock.cc:648] HybridClock initialized: now 1785984713532274 us; error 53 us; skew 500 ppm
I20260806 02:51:53.532685 27292 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:51:53 dist-test-slave-zj8q krb5kdc[27278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984713, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:51:53.535859 27292 init.cc:377] Logged in from keytab as kudu/127.25.211.190@KRBTEST.COM (short username kudu)
I20260806 02:51:53.537359 27292 webserver.cc:460] Webserver started at http://127.25.211.190:34161/ using document root <none> and password file <none>
I20260806 02:51:53.538048 27292 fs_manager.cc:359] Metadata directory not provided
I20260806 02:51:53.538194 27292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:51:53.538513 27292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:51:53.541533 27292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/master-0/data/instance:
uuid: "7d531766747945dfaf692f74a28eb276"
format_stamp: "Formatted at 2026-08-06 02:51:53 on dist-test-slave-zj8q"
server_key: "299dafe8ff6feb6a5ad3fa6b05050988"
server_key_iv: "2e7fdedbb373dd1fa302fe6cd2fedcb6"
server_key_version: "encryptionkey@0"
I20260806 02:51:53.542352 27292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/master-0/wal/instance:
uuid: "7d531766747945dfaf692f74a28eb276"
format_stamp: "Formatted at 2026-08-06 02:51:53 on dist-test-slave-zj8q"
server_key: "299dafe8ff6feb6a5ad3fa6b05050988"
server_key_iv: "2e7fdedbb373dd1fa302fe6cd2fedcb6"
server_key_version: "encryptionkey@0"
I20260806 02:51:53.547829 27292 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 02:51:53.551491 27313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:53.552973 27292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:51:53.553164 27292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/master-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/master-0/wal
uuid: "7d531766747945dfaf692f74a28eb276"
format_stamp: "Formatted at 2026-08-06 02:51:53 on dist-test-slave-zj8q"
server_key: "299dafe8ff6feb6a5ad3fa6b05050988"
server_key_iv: "2e7fdedbb373dd1fa302fe6cd2fedcb6"
server_key_version: "encryptionkey@0"
I20260806 02:51:53.553362 27292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:51:53.573894 27292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:51:53.578585 27292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:51:53.578895 27292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:51:53.598801 27292 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.190:46793
I20260806 02:51:53.598737 27397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.190:46793 every 8 connection(s)
I20260806 02:51:53.600322 27292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
I20260806 02:51:53.602469 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 27292
I20260806 02:51:53.602722 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/master-0/wal/instance
I20260806 02:51:53.603570 26446 external_mini_cluster.cc:1428] Setting key 03b785c2d545c14070f9d0412f2f23a2
I20260806 02:51:53.605495 27399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 06 02:51:53 dist-test-slave-zj8q krb5kdc[27278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984713, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:51:53.620486 27399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d531766747945dfaf692f74a28eb276: Bootstrap starting.
I20260806 02:51:53.624024 27399 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d531766747945dfaf692f74a28eb276: Neither blocks nor log segments found. Creating new log.
I20260806 02:51:53.625341 27399 log.cc:824] T 00000000000000000000000000000000 P 7d531766747945dfaf692f74a28eb276: Log is configured to *not* fsync() on all Append() calls
I20260806 02:51:53.628492 27403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:53.605890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41824 (local address 127.25.211.190:46793)
0806 02:51:53.606756 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:53.606768 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:53.606815 (+ 47us) server_negotiation.cc:407] Connection header received
0806 02:51:53.608488 (+ 1673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:53.608513 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:53.608952 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:53.609316 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:53.611275 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:53.612765 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:53.614586 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:53.615489 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:53.615664 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:53.620922 (+ 5258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:53.620933 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:53.620952 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:53.620997 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:53.624073 (+ 3076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:53.625008 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:53.625015 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:53.625022 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:53.625217 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:53.625648 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:53.625651 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:53.625653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:53.626396 (+ 743us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:53.626730 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:53.627106 (+ 376us) server_negotiation.cc:299] Negotiation successful
0806 02:51:53.627479 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":409,"threads_started":1}
I20260806 02:51:53.628685 27399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d531766747945dfaf692f74a28eb276: No bootstrap required, opened a new log
I20260806 02:51:53.636106 27399 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d531766747945dfaf692f74a28eb276 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d531766747945dfaf692f74a28eb276" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 46793 } }
I20260806 02:51:53.636487 27399 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d531766747945dfaf692f74a28eb276 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:51:53.636593 27399 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d531766747945dfaf692f74a28eb276 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d531766747945dfaf692f74a28eb276, State: Initialized, Role: FOLLOWER
I20260806 02:51:53.637192 27399 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d531766747945dfaf692f74a28eb276 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d531766747945dfaf692f74a28eb276" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 46793 } }
I20260806 02:51:53.637416 27399 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d531766747945dfaf692f74a28eb276 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:51:53.637548 27399 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d531766747945dfaf692f74a28eb276 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:51:53.637705 27399 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d531766747945dfaf692f74a28eb276 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:51:53.639973 27399 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d531766747945dfaf692f74a28eb276 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d531766747945dfaf692f74a28eb276" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 46793 } }
I20260806 02:51:53.640494 27399 leader_election.cc:302] T 00000000000000000000000000000000 P 7d531766747945dfaf692f74a28eb276 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d531766747945dfaf692f74a28eb276; no voters:
I20260806 02:51:53.641047 27399 leader_election.cc:288] T 00000000000000000000000000000000 P 7d531766747945dfaf692f74a28eb276 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:51:53.641173 27405 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d531766747945dfaf692f74a28eb276 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:51:53.642261 27405 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d531766747945dfaf692f74a28eb276 [term 1 LEADER]: Becoming Leader. State: Replica: 7d531766747945dfaf692f74a28eb276, State: Running, Role: LEADER
I20260806 02:51:53.642858 27405 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d531766747945dfaf692f74a28eb276 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d531766747945dfaf692f74a28eb276" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 46793 } }
I20260806 02:51:53.643260 27399 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d531766747945dfaf692f74a28eb276 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:51:53.645882 27407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d531766747945dfaf692f74a28eb276 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d531766747945dfaf692f74a28eb276. Latest consensus state: current_term: 1 leader_uuid: "7d531766747945dfaf692f74a28eb276" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d531766747945dfaf692f74a28eb276" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 46793 } } }
I20260806 02:51:53.646350 27407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d531766747945dfaf692f74a28eb276 [sys.catalog]: This master's current role is: LEADER
I20260806 02:51:53.645797 27406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d531766747945dfaf692f74a28eb276 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d531766747945dfaf692f74a28eb276" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d531766747945dfaf692f74a28eb276" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 46793 } } }
I20260806 02:51:53.646777 27406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d531766747945dfaf692f74a28eb276 [sys.catalog]: This master's current role is: LEADER
I20260806 02:51:53.649402 27418 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:51:53.654377 27418 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:51:53.663966 27418 catalog_manager.cc:1347] Generated new cluster ID: bd5629198d344570862178381a09e045
I20260806 02:51:53.664103 27418 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:51:53.677497 27418 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:51:53.679402 27418 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:51:53.697162 27418 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d531766747945dfaf692f74a28eb276: Generated new TSK 0
I20260806 02:51:53.698071 27418 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.211.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab.
Entry for principal kudu/127.25.211.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab.
I20260806 02:51:53.763917 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.129:0
--local_ip_for_outbound_sockets=127.25.211.129
--webserver_interface=127.25.211.129
--webserver_port=0
--tserver_master_addrs=127.25.211.190:46793
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab
--principal=kudu/127.25.211.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:42205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:51:54.121520 27437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:51:54.121835 27437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:51:54.121933 27437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:51:54.129673 27437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:51:54.129801 27437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:51:54.129923 27437 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.129
I20260806 02:51:54.140149 27437 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:42205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab
--principal=kudu/127.25.211.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.211.129
--webserver_port=0
--tserver_master_addrs=127.25.211.190:46793
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.129
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:51:54.141477 27437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:51:54.142884 27437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:51:54.156534 27443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:54.156765 27444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:54.157160 27446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:54.157467 27437 server_base.cc:1034] running on GCE node
I20260806 02:51:54.158280 27437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:51:54.159335 27437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:51:54.160555 27437 hybrid_clock.cc:648] HybridClock initialized: now 1785984714160487 us; error 59 us; skew 500 ppm
I20260806 02:51:54.160902 27437 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:51:54 dist-test-slave-zj8q krb5kdc[27278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984714, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:51:54.167784 27437 init.cc:377] Logged in from keytab as kudu/127.25.211.129@KRBTEST.COM (short username kudu)
I20260806 02:51:54.169435 27437 webserver.cc:460] Webserver started at http://127.25.211.129:45259/ using document root <none> and password file <none>
I20260806 02:51:54.170202 27437 fs_manager.cc:359] Metadata directory not provided
I20260806 02:51:54.170341 27437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:51:54.170653 27437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:51:54.173640 27437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/ts-0/data/instance:
uuid: "cb092047acf848f6bbdcabf7b94c70f6"
format_stamp: "Formatted at 2026-08-06 02:51:54 on dist-test-slave-zj8q"
server_key: "df3ddc892ca01c28e1d58b115852e37d"
server_key_iv: "42ad74af5bb48efb08ddd4fb9611dd04"
server_key_version: "encryptionkey@0"
I20260806 02:51:54.174507 27437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/ts-0/wal/instance:
uuid: "cb092047acf848f6bbdcabf7b94c70f6"
format_stamp: "Formatted at 2026-08-06 02:51:54 on dist-test-slave-zj8q"
server_key: "df3ddc892ca01c28e1d58b115852e37d"
server_key_iv: "42ad74af5bb48efb08ddd4fb9611dd04"
server_key_version: "encryptionkey@0"
I20260806 02:51:54.179979 27437 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260806 02:51:54.183559 27454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:54.184914 27437 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:51:54.185076 27437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/ts-0/wal
uuid: "cb092047acf848f6bbdcabf7b94c70f6"
format_stamp: "Formatted at 2026-08-06 02:51:54 on dist-test-slave-zj8q"
server_key: "df3ddc892ca01c28e1d58b115852e37d"
server_key_iv: "42ad74af5bb48efb08ddd4fb9611dd04"
server_key_version: "encryptionkey@0"
I20260806 02:51:54.185266 27437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:51:54.221312 27437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:51:54.225553 27437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:51:54.225858 27437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:51:54.226886 27437 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:51:54.228268 27437 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:51:54.228353 27437 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:54.228485 27437 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:51:54.228539 27437 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:54.259976 27437 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.129:39179
I20260806 02:51:54.260020 27623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.129:39179 every 8 connection(s)
I20260806 02:51:54.261229 27437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
I20260806 02:51:54.270215 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 27437
I20260806 02:51:54.270443 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/ts-0/wal/instance
I20260806 02:51:54.271062 26446 external_mini_cluster.cc:1428] Setting key f517f6a3068a3602cbffa13b7278c957
Aug 06 02:51:54 dist-test-slave-zj8q krb5kdc[27278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984714, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.129@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:51:54.282096 27628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:54.263795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:53475 (local address 127.25.211.190:46793)
0806 02:51:54.264372 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:54.264378 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:54.265255 (+ 877us) server_negotiation.cc:407] Connection header received
0806 02:51:54.267250 (+ 1995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:54.267256 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:54.267350 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:54.267450 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:54.269064 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:54.269806 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:54.271599 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:54.272127 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:54.272254 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:54.275965 (+ 3711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:54.275970 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:54.275974 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:54.276010 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:54.278594 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:54.279413 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:54.279417 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:54.279419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:54.279554 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:54.280013 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:54.280017 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:54.280019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:54.280321 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:54.280465 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:54.281303 (+ 838us) server_negotiation.cc:299] Negotiation successful
0806 02:51:54.281578 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":345,"threads_started":1}
I20260806 02:51:54.283231 27629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:54.264385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:46793 (local address 127.25.211.129:53475)
0806 02:51:54.265081 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:54.265126 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:54.266951 (+ 1825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:54.267734 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:54.267742 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:54.268280 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:54.268881 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:54.268901 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:54.269979 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:54.269983 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:54.271443 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:54.271459 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:54.272375 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:54.272385 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:54.272641 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:54.273356 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:54.273388 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:54.275706 (+ 2318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:51:54.278755 (+ 3049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:54.278774 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:54.278790 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:54.279266 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:54.279666 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:54.279671 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:54.279674 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:54.279888 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:54.280458 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:51:54.280466 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:51:54.280865 (+ 399us) client_negotiation.cc:769] Sending connection context
0806 02:51:54.281134 (+ 269us) client_negotiation.cc:240] Negotiation successful
0806 02:51:54.281738 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":260,"threads_started":1}
I20260806 02:51:54.285066 27624 heartbeater.cc:347] Connected to a master server at 127.25.211.190:46793
I20260806 02:51:54.285387 27624 heartbeater.cc:464] Registering TS with master...
I20260806 02:51:54.286265 27624 heartbeater.cc:511] Master 127.25.211.190:46793 requested a full tablet report, sending...
I20260806 02:51:54.288693 27340 ts_manager.cc:194] Registered new tserver with Master: cb092047acf848f6bbdcabf7b94c70f6 (127.25.211.129:39179)
WARNING: no policy specified for kudu/127.25.211.130@KRBTEST.COM; defaulting to no policy
I20260806 02:51:54.290455 27340 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.211.129@KRBTEST.COM'} at 127.25.211.129:53475
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab.
Entry for principal kudu/127.25.211.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab.
I20260806 02:51:54.316752 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.130:0
--local_ip_for_outbound_sockets=127.25.211.130
--webserver_interface=127.25.211.130
--webserver_port=0
--tserver_master_addrs=127.25.211.190:46793
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab
--principal=kudu/127.25.211.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:42205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:51:54.656498 27633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:51:54.656859 27633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:51:54.657006 27633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:51:54.664661 27633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:51:54.664822 27633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:51:54.665014 27633 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.130
I20260806 02:51:54.675314 27633 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:42205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab
--principal=kudu/127.25.211.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.211.130
--webserver_port=0
--tserver_master_addrs=127.25.211.190:46793
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.130
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:51:54.676690 27633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:51:54.678179 27633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:51:54.692395 27642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:54.692368 27643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:54.692350 27646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:54.693236 27633 server_base.cc:1034] running on GCE node
I20260806 02:51:54.693979 27633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:51:54.695200 27633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:51:54.696439 27633 hybrid_clock.cc:648] HybridClock initialized: now 1785984714696369 us; error 67 us; skew 500 ppm
I20260806 02:51:54.696828 27633 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:51:54 dist-test-slave-zj8q krb5kdc[27278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984714, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:51:54.700515 27633 init.cc:377] Logged in from keytab as kudu/127.25.211.130@KRBTEST.COM (short username kudu)
I20260806 02:51:54.702275 27633 webserver.cc:460] Webserver started at http://127.25.211.130:34391/ using document root <none> and password file <none>
I20260806 02:51:54.703033 27633 fs_manager.cc:359] Metadata directory not provided
I20260806 02:51:54.703161 27633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:51:54.703485 27633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:51:54.706696 27633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/ts-1/data/instance:
uuid: "03b778974ec6479e9e4d3e2f42096fc8"
format_stamp: "Formatted at 2026-08-06 02:51:54 on dist-test-slave-zj8q"
server_key: "ec0a43777558752aa6085aa9e1d3a935"
server_key_iv: "a0d55755421cf70a316397a26e085097"
server_key_version: "encryptionkey@0"
I20260806 02:51:54.707513 27633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/ts-1/wal/instance:
uuid: "03b778974ec6479e9e4d3e2f42096fc8"
format_stamp: "Formatted at 2026-08-06 02:51:54 on dist-test-slave-zj8q"
server_key: "ec0a43777558752aa6085aa9e1d3a935"
server_key_iv: "a0d55755421cf70a316397a26e085097"
server_key_version: "encryptionkey@0"
I20260806 02:51:54.713043 27633 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 02:51:54.716739 27658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:54.718111 27633 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260806 02:51:54.718335 27633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/ts-1/wal
uuid: "03b778974ec6479e9e4d3e2f42096fc8"
format_stamp: "Formatted at 2026-08-06 02:51:54 on dist-test-slave-zj8q"
server_key: "ec0a43777558752aa6085aa9e1d3a935"
server_key_iv: "a0d55755421cf70a316397a26e085097"
server_key_version: "encryptionkey@0"
I20260806 02:51:54.718531 27633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:51:54.738626 27633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:51:54.742967 27633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:51:54.743250 27633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:51:54.744375 27633 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:51:54.745882 27633 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:51:54.745977 27633 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:54.746172 27633 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:51:54.746227 27633 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:54.780522 27633 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.130:40195
I20260806 02:51:54.780570 27837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.130:40195 every 8 connection(s)
I20260806 02:51:54.781870 27633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
I20260806 02:51:54.787065 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 27633
I20260806 02:51:54.787269 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/ts-1/wal/instance
I20260806 02:51:54.787693 26446 external_mini_cluster.cc:1428] Setting key c620695d5f725f008c227083cbf9831f
Aug 06 02:51:54 dist-test-slave-zj8q krb5kdc[27278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984714, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.130@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:51:54.804301 27842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:54.784713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:36375 (local address 127.25.211.190:46793)
0806 02:51:54.785219 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:54.785224 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:54.786264 (+ 1040us) server_negotiation.cc:407] Connection header received
0806 02:51:54.788248 (+ 1984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:54.788254 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:54.788341 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:54.788455 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:54.790494 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:54.791519 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:54.793160 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:54.793782 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:54.793913 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:54.798101 (+ 4188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:54.798107 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:54.798110 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:54.798148 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:54.800924 (+ 2776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:54.801823 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:54.801832 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:54.801834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:54.801977 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:54.802471 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:54.802475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:54.802478 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:54.802781 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:54.802922 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:54.803673 (+ 751us) server_negotiation.cc:299] Negotiation successful
0806 02:51:54.803929 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":240,"threads_started":1}
I20260806 02:51:54.805437 27845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:54.785251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:46793 (local address 127.25.211.130:36375)
0806 02:51:54.786059 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:54.786102 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:54.787958 (+ 1856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:54.788709 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:54.788722 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:54.789516 (+ 794us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:54.790299 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:54.790321 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:54.791684 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:54.791688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:54.792988 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:54.793002 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:54.794025 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:54.794030 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:54.794300 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:54.795019 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:54.795050 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:54.797770 (+ 2720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:51:54.801091 (+ 3321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:54.801101 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:54.801119 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:54.801584 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:54.802096 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:54.802101 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:54.802105 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:54.802353 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:54.802910 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:51:54.802920 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:51:54.803303 (+ 383us) client_negotiation.cc:769] Sending connection context
0806 02:51:54.803559 (+ 256us) client_negotiation.cc:240] Negotiation successful
0806 02:51:54.804139 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":337,"threads_started":1}
I20260806 02:51:54.807080 27838 heartbeater.cc:347] Connected to a master server at 127.25.211.190:46793
I20260806 02:51:54.807387 27838 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.25.211.131@KRBTEST.COM; defaulting to no policy
I20260806 02:51:54.808156 27838 heartbeater.cc:511] Master 127.25.211.190:46793 requested a full tablet report, sending...
I20260806 02:51:54.809710 27340 ts_manager.cc:194] Registered new tserver with Master: 03b778974ec6479e9e4d3e2f42096fc8 (127.25.211.130:40195)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.131@KRBTEST.COM" created.
I20260806 02:51:54.810638 27340 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.211.130@KRBTEST.COM'} at 127.25.211.130:36375
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab.
Entry for principal kudu/127.25.211.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab.
I20260806 02:51:54.836174 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.131:0
--local_ip_for_outbound_sockets=127.25.211.131
--webserver_interface=127.25.211.131
--webserver_port=0
--tserver_master_addrs=127.25.211.190:46793
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab
--principal=kudu/127.25.211.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:42205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:51:55.174950 27849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:51:55.175330 27849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:51:55.175485 27849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:51:55.182996 27849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:51:55.183147 27849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:51:55.183320 27849 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.131
I20260806 02:51:55.194034 27849 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:42205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab
--principal=kudu/127.25.211.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.211.131
--webserver_port=0
--tserver_master_addrs=127.25.211.190:46793
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.131
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:51:55.195618 27849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:51:55.197084 27849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:51:55.213842 27857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:55.214741 27858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:55.215468 27862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:55.215945 27849 server_base.cc:1034] running on GCE node
I20260806 02:51:55.216622 27849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:51:55.217607 27849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:51:55.218820 27849 hybrid_clock.cc:648] HybridClock initialized: now 1785984715218760 us; error 56 us; skew 500 ppm
I20260806 02:51:55.219194 27849 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:51:55 dist-test-slave-zj8q krb5kdc[27278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984715, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:51:55.222556 27849 init.cc:377] Logged in from keytab as kudu/127.25.211.131@KRBTEST.COM (short username kudu)
I20260806 02:51:55.224128 27849 webserver.cc:460] Webserver started at http://127.25.211.131:34541/ using document root <none> and password file <none>
I20260806 02:51:55.224879 27849 fs_manager.cc:359] Metadata directory not provided
I20260806 02:51:55.225008 27849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:51:55.225313 27849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:51:55.228513 27849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/ts-2/data/instance:
uuid: "94caff1e6f6e44b0ae78e6c480d6f7cc"
format_stamp: "Formatted at 2026-08-06 02:51:55 on dist-test-slave-zj8q"
server_key: "5bcbcff7a4b45f5069961c8d412599bd"
server_key_iv: "2dd6d9d300b644992b0937a964236e8d"
server_key_version: "encryptionkey@0"
I20260806 02:51:55.229405 27849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/ts-2/wal/instance:
uuid: "94caff1e6f6e44b0ae78e6c480d6f7cc"
format_stamp: "Formatted at 2026-08-06 02:51:55 on dist-test-slave-zj8q"
server_key: "5bcbcff7a4b45f5069961c8d412599bd"
server_key_iv: "2dd6d9d300b644992b0937a964236e8d"
server_key_version: "encryptionkey@0"
I20260806 02:51:55.235353 27849 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260806 02:51:55.239229 27873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:55.240761 27849 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 02:51:55.240965 27849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/ts-2/wal
uuid: "94caff1e6f6e44b0ae78e6c480d6f7cc"
format_stamp: "Formatted at 2026-08-06 02:51:55 on dist-test-slave-zj8q"
server_key: "5bcbcff7a4b45f5069961c8d412599bd"
server_key_iv: "2dd6d9d300b644992b0937a964236e8d"
server_key_version: "encryptionkey@0"
I20260806 02:51:55.241182 27849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:51:55.257998 27849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:51:55.263499 27849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:51:55.263821 27849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:51:55.264915 27849 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:51:55.266479 27849 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:51:55.266561 27849 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:55.266665 27849 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:51:55.266717 27849 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:55.293983 27624 heartbeater.cc:503] Master 127.25.211.190:46793 was elected leader, sending a full tablet report...
I20260806 02:51:55.299191 27849 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.131:45333
I20260806 02:51:55.299234 28057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.131:45333 every 8 connection(s)
I20260806 02:51:55.300616 27849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
I20260806 02:51:55.307691 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 27849
I20260806 02:51:55.307991 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1785984707115702-26446-0/minicluster-data/ts-2/wal/instance
I20260806 02:51:55.308519 26446 external_mini_cluster.cc:1428] Setting key 71e1e5dd8e9e757a43bc36a76b0fb397
Aug 06 02:51:55 dist-test-slave-zj8q krb5kdc[27278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984715, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.131@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:51:55.322489 27842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:55.303343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:34337 (local address 127.25.211.190:46793)
0806 02:51:55.303525 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:55.303531 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:55.304720 (+ 1189us) server_negotiation.cc:407] Connection header received
0806 02:51:55.306705 (+ 1985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:55.306711 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:55.306820 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:55.306911 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:55.308535 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:55.309429 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:55.311499 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:55.311975 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:55.312078 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:55.315754 (+ 3676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:55.315760 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:55.315763 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:55.315798 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:55.319126 (+ 3328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:55.320033 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:55.320038 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:55.320040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:55.320182 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:55.320634 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:55.320638 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:55.320640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:55.320942 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:55.321064 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:55.321877 (+ 813us) server_negotiation.cc:299] Negotiation successful
0806 02:51:55.322113 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 02:51:55.323612 28062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:55.303742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:46793 (local address 127.25.211.131:34337)
0806 02:51:55.304548 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:55.304586 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:55.306413 (+ 1827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:55.307172 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:55.307180 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:55.307708 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:55.308362 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:55.308383 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:55.309630 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:55.309634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:55.311358 (+ 1724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:55.311374 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:55.312167 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:55.312171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:55.312400 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:55.313075 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:55.313098 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:55.315497 (+ 2399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:51:55.319303 (+ 3806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:55.319312 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:55.319328 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:55.319877 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:55.320287 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:55.320290 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:55.320292 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:55.320498 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:55.321078 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:51:55.321085 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:51:55.321460 (+ 375us) client_negotiation.cc:769] Sending connection context
0806 02:51:55.321704 (+ 244us) client_negotiation.cc:240] Negotiation successful
0806 02:51:55.322246 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":327,"threads_started":1}
I20260806 02:51:55.325071 28058 heartbeater.cc:347] Connected to a master server at 127.25.211.190:46793
I20260806 02:51:55.325412 28058 heartbeater.cc:464] Registering TS with master...
I20260806 02:51:55.326341 28058 heartbeater.cc:511] Master 127.25.211.190:46793 requested a full tablet report, sending...
I20260806 02:51:55.327855 27340 ts_manager.cc:194] Registered new tserver with Master: 94caff1e6f6e44b0ae78e6c480d6f7cc (127.25.211.131:45333)
I20260806 02:51:55.328745 27340 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.211.131@KRBTEST.COM'} at 127.25.211.131:34337
I20260806 02:51:55.336872 26446 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:51:55.359863 27842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:55.344085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41832 (local address 127.25.211.190:46793)
0806 02:51:55.344302 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:55.344308 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:55.344804 (+ 496us) server_negotiation.cc:407] Connection header received
0806 02:51:55.345037 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:55.345041 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:55.345126 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:55.345214 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:55.347503 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:55.348277 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:55.349842 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:55.350323 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:55.350436 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:55.353453 (+ 3017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:55.353462 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:55.353467 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:55.353533 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:55.356397 (+ 2864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:55.357309 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:55.357313 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:55.357315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:55.357443 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:55.357938 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:55.357941 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:55.357943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:55.358345 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:55.358551 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:55.359222 (+ 671us) server_negotiation.cc:299] Negotiation successful
0806 02:51:55.359423 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 02:51:55.366958 27340 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41832:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:51:55.375415 27340 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:51:55.401823 28076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:55.388972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:39179 (local address 127.0.0.1:59306)
0806 02:51:55.390220 (+ 1248us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:55.390283 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:55.390476 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:55.393187 (+ 2711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:55.393192 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:55.393217 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:51:55.393873 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:55.393885 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:55.396087 (+ 2202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:55.396092 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:55.398873 (+ 2781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:55.398893 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:55.400530 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:55.400537 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:55.400800 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:55.400843 (+ 43us) client_negotiation.cc:769] Sending connection context
0806 02:51:55.400980 (+ 137us) client_negotiation.cc:240] Negotiation successful
0806 02:51:55.401201 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":937,"mutex_wait_us":7,"thread_start_us":825,"threads_started":1}
I20260806 02:51:55.402390 28078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:55.389567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34338 (local address 127.25.211.130:40195)
0806 02:51:55.392232 (+ 2665us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:55.392240 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:55.392285 (+ 45us) server_negotiation.cc:407] Connection header received
0806 02:51:55.392405 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:55.392413 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:55.392754 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:55.393013 (+ 259us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:51:55.393991 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:55.395376 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:55.398558 (+ 3182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:55.400403 (+ 1845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:55.401591 (+ 1188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:55.401768 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:55.401842 (+ 74us) server_negotiation.cc:299] Negotiation successful
0806 02:51:55.402017 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2473,"thread_start_us":2359,"threads_started":1}
I20260806 02:51:55.402457 28083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:55.389171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.130:40195 (local address 127.0.0.1:34338)
0806 02:51:55.391738 (+ 2567us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:55.391759 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:55.391886 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:55.393032 (+ 1146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:55.393039 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:55.393069 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:51:55.393676 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:55.393693 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:55.395518 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:55.395524 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:55.398335 (+ 2811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:55.398355 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:55.400792 (+ 2437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:55.400801 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:55.401047 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:55.401102 (+ 55us) client_negotiation.cc:769] Sending connection context
0806 02:51:55.401175 (+ 73us) client_negotiation.cc:240] Negotiation successful
0806 02:51:55.401316 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2439,"mutex_wait_us":67,"thread_start_us":3099,"threads_started":1}
I20260806 02:51:55.403213 28077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:55.389876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.131:45333 (local address 127.0.0.1:59800)
0806 02:51:55.392537 (+ 2661us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:55.392555 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:55.392786 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:55.393793 (+ 1007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:55.393801 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:55.393822 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:51:55.394326 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:55.394338 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:55.396423 (+ 2085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:55.396428 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:55.398930 (+ 2502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:55.398944 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:55.401040 (+ 2096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:55.401046 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:55.401252 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:55.401281 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 02:51:55.401371 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 02:51:55.401506 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2556,"mutex_wait_us":601,"thread_start_us":1281,"threads_started":1}
I20260806 02:51:55.403366 28082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:55.389791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59306 (local address 127.25.211.129:39179)
0806 02:51:55.392630 (+ 2839us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:55.392638 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:55.392666 (+ 28us) server_negotiation.cc:407] Connection header received
0806 02:51:55.392775 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:55.392781 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:55.392963 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:55.394082 (+ 1119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:51:55.394341 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:55.395932 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:55.399070 (+ 3138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:55.400321 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:55.400522 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:55.400713 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:55.402806 (+ 2093us) server_negotiation.cc:299] Negotiation successful
0806 02:51:55.402987 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2666,"spinlock_wait_cycles":24832,"thread_start_us":2616,"threads_started":1}
I20260806 02:51:55.404197 28084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:55.390284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59800 (local address 127.25.211.131:45333)
0806 02:51:55.391183 (+ 899us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:55.391193 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:55.392664 (+ 1471us) server_negotiation.cc:407] Connection header received
0806 02:51:55.392976 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:55.392983 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:55.393191 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:55.393431 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:51:55.394540 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:55.396285 (+ 1745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:55.399176 (+ 2891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:55.400823 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:55.401027 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:55.401198 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:55.401454 (+ 256us) server_negotiation.cc:299] Negotiation successful
0806 02:51:55.401724 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":731,"thread_start_us":573,"threads_started":1}
I20260806 02:51:55.406893 27521 tablet_service.cc:1467] Processing CreateTablet for tablet 5f08b5f209ec410ca1c58e7a3616db34 (DEFAULT_TABLE table=test-table [id=dd4b645aafc544299a74adc2a6bfd575]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:51:55.406975 27726 tablet_service.cc:1467] Processing CreateTablet for tablet 5f08b5f209ec410ca1c58e7a3616db34 (DEFAULT_TABLE table=test-table [id=dd4b645aafc544299a74adc2a6bfd575]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:51:55.408571 27726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f08b5f209ec410ca1c58e7a3616db34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:51:55.408609 27521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f08b5f209ec410ca1c58e7a3616db34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:51:55.408423 27942 tablet_service.cc:1467] Processing CreateTablet for tablet 5f08b5f209ec410ca1c58e7a3616db34 (DEFAULT_TABLE table=test-table [id=dd4b645aafc544299a74adc2a6bfd575]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:51:55.409863 27942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f08b5f209ec410ca1c58e7a3616db34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:51:55.420595 28085 tablet_bootstrap.cc:492] T 5f08b5f209ec410ca1c58e7a3616db34 P cb092047acf848f6bbdcabf7b94c70f6: Bootstrap starting.
I20260806 02:51:55.424849 28086 tablet_bootstrap.cc:492] T 5f08b5f209ec410ca1c58e7a3616db34 P 03b778974ec6479e9e4d3e2f42096fc8: Bootstrap starting.
I20260806 02:51:55.425977 28085 tablet_bootstrap.cc:654] T 5f08b5f209ec410ca1c58e7a3616db34 P cb092047acf848f6bbdcabf7b94c70f6: Neither blocks nor log segments found. Creating new log.
I20260806 02:51:55.427407 28085 log.cc:824] T 5f08b5f209ec410ca1c58e7a3616db34 P cb092047acf848f6bbdcabf7b94c70f6: Log is configured to *not* fsync() on all Append() calls
I20260806 02:51:55.429049 28086 tablet_bootstrap.cc:654] T 5f08b5f209ec410ca1c58e7a3616db34 P 03b778974ec6479e9e4d3e2f42096fc8: Neither blocks nor log segments found. Creating new log.
I20260806 02:51:55.429157 28087 tablet_bootstrap.cc:492] T 5f08b5f209ec410ca1c58e7a3616db34 P 94caff1e6f6e44b0ae78e6c480d6f7cc: Bootstrap starting.
I20260806 02:51:55.430541 28086 log.cc:824] T 5f08b5f209ec410ca1c58e7a3616db34 P 03b778974ec6479e9e4d3e2f42096fc8: Log is configured to *not* fsync() on all Append() calls
I20260806 02:51:55.431915 28085 tablet_bootstrap.cc:492] T 5f08b5f209ec410ca1c58e7a3616db34 P cb092047acf848f6bbdcabf7b94c70f6: No bootstrap required, opened a new log
I20260806 02:51:55.432194 28085 ts_tablet_manager.cc:1397] T 5f08b5f209ec410ca1c58e7a3616db34 P cb092047acf848f6bbdcabf7b94c70f6: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260806 02:51:55.433879 28087 tablet_bootstrap.cc:654] T 5f08b5f209ec410ca1c58e7a3616db34 P 94caff1e6f6e44b0ae78e6c480d6f7cc: Neither blocks nor log segments found. Creating new log.
I20260806 02:51:55.434628 28086 tablet_bootstrap.cc:492] T 5f08b5f209ec410ca1c58e7a3616db34 P 03b778974ec6479e9e4d3e2f42096fc8: No bootstrap required, opened a new log
I20260806 02:51:55.434921 28086 ts_tablet_manager.cc:1397] T 5f08b5f209ec410ca1c58e7a3616db34 P 03b778974ec6479e9e4d3e2f42096fc8: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260806 02:51:55.435071 28087 log.cc:824] T 5f08b5f209ec410ca1c58e7a3616db34 P 94caff1e6f6e44b0ae78e6c480d6f7cc: Log is configured to *not* fsync() on all Append() calls
I20260806 02:51:55.437970 28087 tablet_bootstrap.cc:492] T 5f08b5f209ec410ca1c58e7a3616db34 P 94caff1e6f6e44b0ae78e6c480d6f7cc: No bootstrap required, opened a new log
I20260806 02:51:55.438292 28087 ts_tablet_manager.cc:1397] T 5f08b5f209ec410ca1c58e7a3616db34 P 94caff1e6f6e44b0ae78e6c480d6f7cc: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.005s
I20260806 02:51:55.443100 28085 raft_consensus.cc:348] T 5f08b5f209ec410ca1c58e7a3616db34 P cb092047acf848f6bbdcabf7b94c70f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb092047acf848f6bbdcabf7b94c70f6" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 39179 } } peers { permanent_uuid: "03b778974ec6479e9e4d3e2f42096fc8" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40195 } } peers { permanent_uuid: "94caff1e6f6e44b0ae78e6c480d6f7cc" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45333 } }
I20260806 02:51:55.443678 28085 raft_consensus.cc:374] T 5f08b5f209ec410ca1c58e7a3616db34 P cb092047acf848f6bbdcabf7b94c70f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:51:55.443933 28085 raft_consensus.cc:729] T 5f08b5f209ec410ca1c58e7a3616db34 P cb092047acf848f6bbdcabf7b94c70f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb092047acf848f6bbdcabf7b94c70f6, State: Initialized, Role: FOLLOWER
I20260806 02:51:55.444979 28085 consensus_queue.cc:260] T 5f08b5f209ec410ca1c58e7a3616db34 P cb092047acf848f6bbdcabf7b94c70f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb092047acf848f6bbdcabf7b94c70f6" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 39179 } } peers { permanent_uuid: "03b778974ec6479e9e4d3e2f42096fc8" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40195 } } peers { permanent_uuid: "94caff1e6f6e44b0ae78e6c480d6f7cc" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45333 } }
I20260806 02:51:55.447489 28085 ts_tablet_manager.cc:1428] T 5f08b5f209ec410ca1c58e7a3616db34 P cb092047acf848f6bbdcabf7b94c70f6: Time spent starting tablet: real 0.015s user 0.015s sys 0.001s
I20260806 02:51:55.447449 28086 raft_consensus.cc:348] T 5f08b5f209ec410ca1c58e7a3616db34 P 03b778974ec6479e9e4d3e2f42096fc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb092047acf848f6bbdcabf7b94c70f6" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 39179 } } peers { permanent_uuid: "03b778974ec6479e9e4d3e2f42096fc8" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40195 } } peers { permanent_uuid: "94caff1e6f6e44b0ae78e6c480d6f7cc" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45333 } }
I20260806 02:51:55.448237 28086 raft_consensus.cc:374] T 5f08b5f209ec410ca1c58e7a3616db34 P 03b778974ec6479e9e4d3e2f42096fc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:51:55.448367 28086 raft_consensus.cc:729] T 5f08b5f209ec410ca1c58e7a3616db34 P 03b778974ec6479e9e4d3e2f42096fc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03b778974ec6479e9e4d3e2f42096fc8, State: Initialized, Role: FOLLOWER
I20260806 02:51:55.448608 28087 raft_consensus.cc:348] T 5f08b5f209ec410ca1c58e7a3616db34 P 94caff1e6f6e44b0ae78e6c480d6f7cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb092047acf848f6bbdcabf7b94c70f6" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 39179 } } peers { permanent_uuid: "03b778974ec6479e9e4d3e2f42096fc8" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40195 } } peers { permanent_uuid: "94caff1e6f6e44b0ae78e6c480d6f7cc" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45333 } }
I20260806 02:51:55.449057 28087 raft_consensus.cc:374] T 5f08b5f209ec410ca1c58e7a3616db34 P 94caff1e6f6e44b0ae78e6c480d6f7cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:51:55.449170 28087 raft_consensus.cc:729] T 5f08b5f209ec410ca1c58e7a3616db34 P 94caff1e6f6e44b0ae78e6c480d6f7cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94caff1e6f6e44b0ae78e6c480d6f7cc, State: Initialized, Role: FOLLOWER
I20260806 02:51:55.449057 28086 consensus_queue.cc:260] T 5f08b5f209ec410ca1c58e7a3616db34 P 03b778974ec6479e9e4d3e2f42096fc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb092047acf848f6bbdcabf7b94c70f6" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 39179 } } peers { permanent_uuid: "03b778974ec6479e9e4d3e2f42096fc8" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40195 } } peers { permanent_uuid: "94caff1e6f6e44b0ae78e6c480d6f7cc" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45333 } }
I20260806 02:51:55.449790 28087 consensus_queue.cc:260] T 5f08b5f209ec410ca1c58e7a3616db34 P 94caff1e6f6e44b0ae78e6c480d6f7cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb092047acf848f6bbdcabf7b94c70f6" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 39179 } } peers { permanent_uuid: "03b778974ec6479e9e4d3e2f42096fc8" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40195 } } peers { permanent_uuid: "94caff1e6f6e44b0ae78e6c480d6f7cc" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45333 } }
I20260806 02:51:55.450529 27838 heartbeater.cc:503] Master 127.25.211.190:46793 was elected leader, sending a full tablet report...
I20260806 02:51:55.451159 28086 ts_tablet_manager.cc:1428] T 5f08b5f209ec410ca1c58e7a3616db34 P 03b778974ec6479e9e4d3e2f42096fc8: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260806 02:51:55.452308 28058 heartbeater.cc:503] Master 127.25.211.190:46793 was elected leader, sending a full tablet report...
I20260806 02:51:55.453576 28087 ts_tablet_manager.cc:1428] T 5f08b5f209ec410ca1c58e7a3616db34 P 94caff1e6f6e44b0ae78e6c480d6f7cc: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
W20260806 02:51:55.513674 27625 tablet.cc:2369] T 5f08b5f209ec410ca1c58e7a3616db34 P cb092047acf848f6bbdcabf7b94c70f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:51:55.534063 27839 tablet.cc:2369] T 5f08b5f209ec410ca1c58e7a3616db34 P 03b778974ec6479e9e4d3e2f42096fc8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:51:55.552472 28059 tablet.cc:2369] T 5f08b5f209ec410ca1c58e7a3616db34 P 94caff1e6f6e44b0ae78e6c480d6f7cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:51:55.617522 28098 raft_consensus.cc:482] T 5f08b5f209ec410ca1c58e7a3616db34 P 94caff1e6f6e44b0ae78e6c480d6f7cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:51:55.617899 28098 raft_consensus.cc:504] T 5f08b5f209ec410ca1c58e7a3616db34 P 94caff1e6f6e44b0ae78e6c480d6f7cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb092047acf848f6bbdcabf7b94c70f6" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 39179 } } peers { permanent_uuid: "03b778974ec6479e9e4d3e2f42096fc8" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40195 } } peers { permanent_uuid: "94caff1e6f6e44b0ae78e6c480d6f7cc" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45333 } }
I20260806 02:51:55.619969 28098 leader_election.cc:288] T 5f08b5f209ec410ca1c58e7a3616db34 P 94caff1e6f6e44b0ae78e6c480d6f7cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb092047acf848f6bbdcabf7b94c70f6 (127.25.211.129:39179), 03b778974ec6479e9e4d3e2f42096fc8 (127.25.211.130:40195)
I20260806 02:51:55.627280 28082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:55.620316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:58997 (local address 127.25.211.129:39179)
0806 02:51:55.620564 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:55.620571 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:55.621014 (+ 443us) server_negotiation.cc:407] Connection header received
0806 02:51:55.621220 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:55.621223 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:55.621295 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:55.621388 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:51:55.622180 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:55.623296 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:55.625428 (+ 2132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:55.626388 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:55.626539 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:55.626617 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:55.626893 (+ 276us) server_negotiation.cc:299] Negotiation successful
0806 02:51:55.627026 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260806 02:51:55.627321 28062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:55.620609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:39179 (local address 127.25.211.131:58997)
0806 02:51:55.620888 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:55.620913 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:55.621081 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:55.621451 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:55.621457 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:55.621493 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:51:55.621927 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:55.621946 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:55.623413 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:55.623419 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:55.625233 (+ 1814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:55.625252 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:55.626539 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:55.626545 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:55.626739 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:55.626765 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 02:51:55.626849 (+ 84us) client_negotiation.cc:240] Negotiation successful
0806 02:51:55.627004 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74,"mutex_wait_us":66}
I20260806 02:51:55.628018 28078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:55.620701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:33195 (local address 127.25.211.130:40195)
0806 02:51:55.620922 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:55.620927 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:55.622211 (+ 1284us) server_negotiation.cc:407] Connection header received
0806 02:51:55.622417 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:55.622421 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:55.622483 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:55.622555 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:51:55.623340 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:55.624250 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:55.626224 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:55.627024 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:55.627173 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:55.627244 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:55.627632 (+ 388us) server_negotiation.cc:299] Negotiation successful
0806 02:51:55.627771 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260806 02:51:55.628433 27554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f08b5f209ec410ca1c58e7a3616db34" candidate_uuid: "94caff1e6f6e44b0ae78e6c480d6f7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb092047acf848f6bbdcabf7b94c70f6" is_pre_election: true
I20260806 02:51:55.628744 28099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:55.621020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.130:40195 (local address 127.25.211.131:33195)
0806 02:51:55.622084 (+ 1064us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:55.622105 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:55.622286 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:55.622618 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:55.622623 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:55.622650 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:51:55.623119 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:55.623133 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:55.624422 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:55.624432 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:55.626032 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:55.626045 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:55.627259 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:55.627265 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:55.627437 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:55.627474 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 02:51:55.628226 (+ 752us) client_negotiation.cc:240] Negotiation successful
0806 02:51:55.628369 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":935,"mutex_wait_us":4,"thread_start_us":619,"threads_started":1}
I20260806 02:51:55.629122 27554 raft_consensus.cc:2454] T 5f08b5f209ec410ca1c58e7a3616db34 P cb092047acf848f6bbdcabf7b94c70f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94caff1e6f6e44b0ae78e6c480d6f7cc in term 0.
I20260806 02:51:55.629908 27878 leader_election.cc:302] T 5f08b5f209ec410ca1c58e7a3616db34 P 94caff1e6f6e44b0ae78e6c480d6f7cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94caff1e6f6e44b0ae78e6c480d6f7cc, cb092047acf848f6bbdcabf7b94c70f6; no voters:
I20260806 02:51:55.629912 27758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f08b5f209ec410ca1c58e7a3616db34" candidate_uuid: "94caff1e6f6e44b0ae78e6c480d6f7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03b778974ec6479e9e4d3e2f42096fc8" is_pre_election: true
I20260806 02:51:55.630439 28098 raft_consensus.cc:2781] T 5f08b5f209ec410ca1c58e7a3616db34 P 94caff1e6f6e44b0ae78e6c480d6f7cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:51:55.630537 27758 raft_consensus.cc:2454] T 5f08b5f209ec410ca1c58e7a3616db34 P 03b778974ec6479e9e4d3e2f42096fc8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94caff1e6f6e44b0ae78e6c480d6f7cc in term 0.
I20260806 02:51:55.630648 28098 raft_consensus.cc:482] T 5f08b5f209ec410ca1c58e7a3616db34 P 94caff1e6f6e44b0ae78e6c480d6f7cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:51:55.630769 28098 raft_consensus.cc:3037] T 5f08b5f209ec410ca1c58e7a3616db34 P 94caff1e6f6e44b0ae78e6c480d6f7cc [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:51:55.633659 28098 raft_consensus.cc:504] T 5f08b5f209ec410ca1c58e7a3616db34 P 94caff1e6f6e44b0ae78e6c480d6f7cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb092047acf848f6bbdcabf7b94c70f6" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 39179 } } peers { permanent_uuid: "03b778974ec6479e9e4d3e2f42096fc8" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40195 } } peers { permanent_uuid: "94caff1e6f6e44b0ae78e6c480d6f7cc" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45333 } }
I20260806 02:51:55.634322 28098 leader_election.cc:288] T 5f08b5f209ec410ca1c58e7a3616db34 P 94caff1e6f6e44b0ae78e6c480d6f7cc [CANDIDATE]: Term 1 election: Requested vote from peers cb092047acf848f6bbdcabf7b94c70f6 (127.25.211.129:39179), 03b778974ec6479e9e4d3e2f42096fc8 (127.25.211.130:40195)
I20260806 02:51:55.634786 27758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f08b5f209ec410ca1c58e7a3616db34" candidate_uuid: "94caff1e6f6e44b0ae78e6c480d6f7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03b778974ec6479e9e4d3e2f42096fc8"
I20260806 02:51:55.634893 27554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f08b5f209ec410ca1c58e7a3616db34" candidate_uuid: "94caff1e6f6e44b0ae78e6c480d6f7cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb092047acf848f6bbdcabf7b94c70f6"
I20260806 02:51:55.634971 27758 raft_consensus.cc:3037] T 5f08b5f209ec410ca1c58e7a3616db34 P 03b778974ec6479e9e4d3e2f42096fc8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:51:55.635118 27554 raft_consensus.cc:3037] T 5f08b5f209ec410ca1c58e7a3616db34 P cb092047acf848f6bbdcabf7b94c70f6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:51:55.637292 27758 raft_consensus.cc:2454] T 5f08b5f209ec410ca1c58e7a3616db34 P 03b778974ec6479e9e4d3e2f42096fc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94caff1e6f6e44b0ae78e6c480d6f7cc in term 1.
I20260806 02:51:55.637668 27554 raft_consensus.cc:2454] T 5f08b5f209ec410ca1c58e7a3616db34 P cb092047acf848f6bbdcabf7b94c70f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94caff1e6f6e44b0ae78e6c480d6f7cc in term 1.
I20260806 02:51:55.637739 27875 leader_election.cc:302] T 5f08b5f209ec410ca1c58e7a3616db34 P 94caff1e6f6e44b0ae78e6c480d6f7cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03b778974ec6479e9e4d3e2f42096fc8, 94caff1e6f6e44b0ae78e6c480d6f7cc; no voters:
I20260806 02:51:55.638073 28098 raft_consensus.cc:2781] T 5f08b5f209ec410ca1c58e7a3616db34 P 94caff1e6f6e44b0ae78e6c480d6f7cc [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:51:55.638614 28098 raft_consensus.cc:686] T 5f08b5f209ec410ca1c58e7a3616db34 P 94caff1e6f6e44b0ae78e6c480d6f7cc [term 1 LEADER]: Becoming Leader. State: Replica: 94caff1e6f6e44b0ae78e6c480d6f7cc, State: Running, Role: LEADER
I20260806 02:51:55.639184 28098 consensus_queue.cc:237] T 5f08b5f209ec410ca1c58e7a3616db34 P 94caff1e6f6e44b0ae78e6c480d6f7cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb092047acf848f6bbdcabf7b94c70f6" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 39179 } } peers { permanent_uuid: "03b778974ec6479e9e4d3e2f42096fc8" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40195 } } peers { permanent_uuid: "94caff1e6f6e44b0ae78e6c480d6f7cc" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45333 } }
I20260806 02:51:55.644030 27338 catalog_manager.cc:5495] T 5f08b5f209ec410ca1c58e7a3616db34 P 94caff1e6f6e44b0ae78e6c480d6f7cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 94caff1e6f6e44b0ae78e6c480d6f7cc (127.25.211.131). New cstate: current_term: 1 leader_uuid: "94caff1e6f6e44b0ae78e6c480d6f7cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb092047acf848f6bbdcabf7b94c70f6" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 39179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03b778974ec6479e9e4d3e2f42096fc8" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "94caff1e6f6e44b0ae78e6c480d6f7cc" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45333 } health_report { overall_health: HEALTHY } } }
Aug 06 02:51:55 dist-test-slave-zj8q krb5kdc[27278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984713, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.129@KRBTEST.COM
I20260806 02:51:55.708135 28082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:55.690843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59316 (local address 127.25.211.129:39179)
0806 02:51:55.691023 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:55.691028 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:55.691878 (+ 850us) server_negotiation.cc:407] Connection header received
0806 02:51:55.692012 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:55.692016 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:55.692092 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:55.692179 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:55.694201 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:55.695025 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:55.696504 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:55.696951 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:55.697079 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:55.701763 (+ 4684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:55.701781 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:55.701800 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:55.701848 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:55.705230 (+ 3382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:55.706016 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:55.706025 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:55.706027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:55.706241 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:55.706630 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:55.706635 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:55.706637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:55.707037 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:55.707272 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:55.707562 (+ 290us) server_negotiation.cc:299] Negotiation successful
0806 02:51:55.707769 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Aug 06 02:51:55 dist-test-slave-zj8q krb5kdc[27278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984715, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 02:51:55 dist-test-slave-zj8q krb5kdc[27278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984715, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.25.211.129@KRBTEST.COM
I20260806 02:51:55.764067 28082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:55.748388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59330 (local address 127.25.211.129:39179)
0806 02:51:55.748647 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:55.748655 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:55.749018 (+ 363us) server_negotiation.cc:407] Connection header received
0806 02:51:55.749213 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:55.749217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:55.749285 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:55.749371 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:55.750938 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:55.751696 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:55.752965 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:55.753566 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:55.753710 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:55.758366 (+ 4656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:55.758372 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:55.758376 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:55.758417 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:55.761348 (+ 2931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:55.762025 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:55.762029 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:55.762032 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:55.762202 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:55.762619 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:55.762624 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:55.762628 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:55.762923 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:55.763210 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:55.763472 (+ 262us) server_negotiation.cc:299] Negotiation successful
0806 02:51:55.763731 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
Aug 06 02:51:55 dist-test-slave-zj8q krb5kdc[27278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 02:51:55 dist-test-slave-zj8q krb5kdc[27278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.129@KRBTEST.COM
I20260806 02:51:55.819845 28082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:55.804632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59338 (local address 127.25.211.129:39179)
0806 02:51:55.804812 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:55.804820 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:55.805170 (+ 350us) server_negotiation.cc:407] Connection header received
0806 02:51:55.805326 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:55.805330 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:55.805398 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:55.805483 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:55.806942 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:55.807664 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:55.809109 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:55.809536 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:55.809642 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:55.813975 (+ 4333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:55.813981 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:55.813985 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:55.814025 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:55.816843 (+ 2818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:55.817670 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:55.817675 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:55.817678 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:55.817832 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:55.818254 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:55.818258 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:55.818260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:55.818597 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:55.818881 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:55.819195 (+ 314us) server_negotiation.cc:299] Negotiation successful
0806 02:51:55.819450 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260806 02:51:55.822211 27494 tablet_service.cc:2908] Rejecting scan request for tablet 5f08b5f209ec410ca1c58e7a3616db34: Uninitialized: clean time has not yet been initialized
I20260806 02:51:55.826148 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 27437
I20260806 02:51:55.840305 28082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:55.827199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59340 (local address 127.25.211.129:39179)
0806 02:51:55.827453 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:55.827460 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:55.827644 (+ 184us) server_negotiation.cc:407] Connection header received
0806 02:51:55.827771 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:55.827775 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:55.827846 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:55.827933 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:55.829150 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:55.829934 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:55.831253 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:55.831701 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:55.831835 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:55.834512 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:55.834517 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:55.834521 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:55.834560 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:55.837392 (+ 2832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:55.838222 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:55.838227 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:55.838230 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:55.838368 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:55.838838 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:55.838842 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:55.838844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:55.839165 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:55.839418 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:55.839807 (+ 389us) server_negotiation.cc:299] Negotiation successful
0806 02:51:55.840000 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260806 02:51:55.841056 27619 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:51:55.932540 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 27437
I20260806 02:51:55.958387 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 27633
Aug 06 02:51:55 dist-test-slave-zj8q krb5kdc[27278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.130@KRBTEST.COM
I20260806 02:51:55.975044 28078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:55.959253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34342 (local address 127.25.211.130:40195)
0806 02:51:55.959441 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:55.959447 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:55.959469 (+ 22us) server_negotiation.cc:407] Connection header received
0806 02:51:55.959588 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:55.959593 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:55.959672 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:55.959780 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:55.960988 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:55.961728 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:55.963174 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:55.963580 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:55.963758 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:55.968506 (+ 4748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:55.968523 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:55.968544 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:55.968590 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:55.971907 (+ 3317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:55.972772 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:55.972781 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:55.972783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:55.972955 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:55.973344 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:55.973348 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:55.973351 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:55.973788 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:55.974039 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:55.974541 (+ 502us) server_negotiation.cc:299] Negotiation successful
0806 02:51:55.974752 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 02:51:55.975689 27830 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:51:56.071774 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 27633
W20260806 02:51:56.091482 27875 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.211.130:40195 (error 108)
I20260806 02:51:56.091799 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 27849
I20260806 02:51:56.093256 28099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:56.092712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.130:40195 (local address 127.25.211.131:33807)
0806 02:51:56.092958 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:56.093093 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.130:40195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260806 02:51:56.093475 27875 consensus_peers.cc:487] T 5f08b5f209ec410ca1c58e7a3616db34 P 94caff1e6f6e44b0ae78e6c480d6f7cc -> Peer 03b778974ec6479e9e4d3e2f42096fc8 (127.25.211.130:40195): Couldn't send request to peer 03b778974ec6479e9e4d3e2f42096fc8. Status: Network error: Client connection negotiation failed: client connection to 127.25.211.130:40195: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
Aug 06 02:51:56 dist-test-slave-zj8q krb5kdc[27278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.131@KRBTEST.COM
I20260806 02:51:56.107287 28130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:56.092558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59804 (local address 127.25.211.131:45333)
0806 02:51:56.093218 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:56.093224 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:56.093258 (+ 34us) server_negotiation.cc:407] Connection header received
0806 02:51:56.093311 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:56.093316 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:56.093397 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:56.093507 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:56.094856 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:56.095571 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:56.096931 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:56.097355 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:56.097450 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:56.101576 (+ 4126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:56.101591 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:56.101608 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:56.101645 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:56.104521 (+ 2876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:56.105221 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:56.105229 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:56.105231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:56.105382 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:56.105759 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:56.105763 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:56.105765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:56.106105 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:56.106397 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:56.106700 (+ 303us) server_negotiation.cc:299] Negotiation successful
0806 02:51:56.106920 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":456,"threads_started":1}
I20260806 02:51:56.107993 28050 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:51:56.197899 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 27849
I20260806 02:51:56.219798 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 27292
I20260806 02:51:56.220989 27388 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:51:56.332032 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 27292
2026-08-06T02:51:56Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4598 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:51:56 dist-test-slave-zj8q krb5kdc[28137](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:51:56 dist-test-slave-zj8q krb5kdc[28137](info): set up 2 sockets
Aug 06 02:51:56 dist-test-slave-zj8q krb5kdc[28137](info): commencing operation
krb5kdc: starting...
W20260806 02:51:57.528785 26446 mini_kdc.cc:121] Time spent starting KDC: real 1.147s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Aug 06 02:51:57 dist-test-slave-zj8q krb5kdc[28137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:51:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:51:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.211.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab.
Entry for principal kudu/127.25.211.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab.
I20260806 02:51:57.746443 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:41109
--webserver_interface=127.25.211.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:35545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.211.190:41109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:51:58.095247 28155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:51:58.095594 28155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:51:58.095703 28155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:51:58.103246 28155 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:51:58.103368 28155 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 02:51:58.103434 28155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:51:58.103523 28155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:51:58.103583 28155 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:51:58.103670 28155 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:51:58.114305 28155 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:35545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.211.190:41109
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:41109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.211.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:51:58.115775 28155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:51:58.117270 28155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:51:58.129936 28164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:58.129992 28166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:58.129906 28171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:58.130868 28155 server_base.cc:1034] running on GCE node
I20260806 02:51:58.131735 28155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:51:58.133253 28155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:51:58.134480 28155 hybrid_clock.cc:648] HybridClock initialized: now 1785984718134410 us; error 36 us; skew 500 ppm
I20260806 02:51:58.134828 28155 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:51:58 dist-test-slave-zj8q krb5kdc[28137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984718, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:51:58.138032 28155 init.cc:377] Logged in from keytab as kudu/127.25.211.190@KRBTEST.COM (short username kudu)
I20260806 02:51:58.139622 28155 webserver.cc:460] Webserver started at http://127.25.211.190:46447/ using document root <none> and password file <none>
I20260806 02:51:58.140352 28155 fs_manager.cc:359] Metadata directory not provided
I20260806 02:51:58.140458 28155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:51:58.140780 28155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:51:58.143832 28155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/master-0/data/instance:
uuid: "db26e8a240ca4210a0c3a59bf501597c"
format_stamp: "Formatted at 2026-08-06 02:51:58 on dist-test-slave-zj8q"
server_key: "c10623468224aabb5fea68ac08455f27"
server_key_iv: "d4983b19121c426c52c54717443a06c6"
server_key_version: "encryptionkey@0"
I20260806 02:51:58.144605 28155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/master-0/wal/instance:
uuid: "db26e8a240ca4210a0c3a59bf501597c"
format_stamp: "Formatted at 2026-08-06 02:51:58 on dist-test-slave-zj8q"
server_key: "c10623468224aabb5fea68ac08455f27"
server_key_iv: "d4983b19121c426c52c54717443a06c6"
server_key_version: "encryptionkey@0"
I20260806 02:51:58.150089 28155 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260806 02:51:58.154179 28187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:58.155659 28155 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:51:58.155856 28155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/master-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/master-0/wal
uuid: "db26e8a240ca4210a0c3a59bf501597c"
format_stamp: "Formatted at 2026-08-06 02:51:58 on dist-test-slave-zj8q"
server_key: "c10623468224aabb5fea68ac08455f27"
server_key_iv: "d4983b19121c426c52c54717443a06c6"
server_key_version: "encryptionkey@0"
I20260806 02:51:58.156103 28155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:51:58.175021 28155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:51:58.179757 28155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:51:58.180038 28155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:51:58.202466 28155 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.190:41109
I20260806 02:51:58.202472 28273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.190:41109 every 8 connection(s)
I20260806 02:51:58.204090 28155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
I20260806 02:51:58.208241 28275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:51:58.208976 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 28155
I20260806 02:51:58.209230 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/master-0/wal/instance
I20260806 02:51:58.210098 26446 external_mini_cluster.cc:1428] Setting key eb2c096ca80e809175c04286226f750d
I20260806 02:51:58.223567 28275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db26e8a240ca4210a0c3a59bf501597c: Bootstrap starting.
Aug 06 02:51:58 dist-test-slave-zj8q krb5kdc[28137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984717, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:51:58.227664 28275 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db26e8a240ca4210a0c3a59bf501597c: Neither blocks nor log segments found. Creating new log.
I20260806 02:51:58.228994 28275 log.cc:824] T 00000000000000000000000000000000 P db26e8a240ca4210a0c3a59bf501597c: Log is configured to *not* fsync() on all Append() calls
I20260806 02:51:58.232885 28275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db26e8a240ca4210a0c3a59bf501597c: No bootstrap required, opened a new log
I20260806 02:51:58.236068 28282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:58.212457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44768 (local address 127.25.211.190:41109)
0806 02:51:58.213361 (+ 904us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:58.213374 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:58.213424 (+ 50us) server_negotiation.cc:407] Connection header received
0806 02:51:58.215395 (+ 1971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:58.215424 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:58.215916 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:58.216301 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:58.218195 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:58.219584 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:58.221368 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:58.222084 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:58.222277 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:58.227239 (+ 4962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:58.227248 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:58.227267 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:58.227311 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:58.231107 (+ 3796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:58.232215 (+ 1108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:58.232223 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:58.232231 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:58.232439 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:58.233066 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:58.233072 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:58.233075 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:58.233700 (+ 625us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:58.233939 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:58.234475 (+ 536us) server_negotiation.cc:299] Negotiation successful
0806 02:51:58.234878 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":426,"threads_started":1}
I20260806 02:51:58.241634 28275 raft_consensus.cc:348] T 00000000000000000000000000000000 P db26e8a240ca4210a0c3a59bf501597c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db26e8a240ca4210a0c3a59bf501597c" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 41109 } }
I20260806 02:51:58.242025 28275 raft_consensus.cc:374] T 00000000000000000000000000000000 P db26e8a240ca4210a0c3a59bf501597c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:51:58.242142 28275 raft_consensus.cc:729] T 00000000000000000000000000000000 P db26e8a240ca4210a0c3a59bf501597c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db26e8a240ca4210a0c3a59bf501597c, State: Initialized, Role: FOLLOWER
I20260806 02:51:58.242748 28275 consensus_queue.cc:260] T 00000000000000000000000000000000 P db26e8a240ca4210a0c3a59bf501597c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db26e8a240ca4210a0c3a59bf501597c" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 41109 } }
I20260806 02:51:58.242972 28275 raft_consensus.cc:388] T 00000000000000000000000000000000 P db26e8a240ca4210a0c3a59bf501597c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:51:58.243119 28275 raft_consensus.cc:482] T 00000000000000000000000000000000 P db26e8a240ca4210a0c3a59bf501597c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:51:58.243283 28275 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db26e8a240ca4210a0c3a59bf501597c [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:51:58.245535 28275 raft_consensus.cc:504] T 00000000000000000000000000000000 P db26e8a240ca4210a0c3a59bf501597c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db26e8a240ca4210a0c3a59bf501597c" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 41109 } }
I20260806 02:51:58.246111 28275 leader_election.cc:302] T 00000000000000000000000000000000 P db26e8a240ca4210a0c3a59bf501597c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db26e8a240ca4210a0c3a59bf501597c; no voters:
I20260806 02:51:58.246778 28275 leader_election.cc:288] T 00000000000000000000000000000000 P db26e8a240ca4210a0c3a59bf501597c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:51:58.246918 28288 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db26e8a240ca4210a0c3a59bf501597c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:51:58.247607 28288 raft_consensus.cc:686] T 00000000000000000000000000000000 P db26e8a240ca4210a0c3a59bf501597c [term 1 LEADER]: Becoming Leader. State: Replica: db26e8a240ca4210a0c3a59bf501597c, State: Running, Role: LEADER
I20260806 02:51:58.248193 28288 consensus_queue.cc:237] T 00000000000000000000000000000000 P db26e8a240ca4210a0c3a59bf501597c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db26e8a240ca4210a0c3a59bf501597c" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 41109 } }
I20260806 02:51:58.248895 28275 sys_catalog.cc:564] T 00000000000000000000000000000000 P db26e8a240ca4210a0c3a59bf501597c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:51:58.251286 28290 sys_catalog.cc:455] T 00000000000000000000000000000000 P db26e8a240ca4210a0c3a59bf501597c [sys.catalog]: SysCatalogTable state changed. Reason: New leader db26e8a240ca4210a0c3a59bf501597c. Latest consensus state: current_term: 1 leader_uuid: "db26e8a240ca4210a0c3a59bf501597c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db26e8a240ca4210a0c3a59bf501597c" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 41109 } } }
I20260806 02:51:58.251755 28290 sys_catalog.cc:458] T 00000000000000000000000000000000 P db26e8a240ca4210a0c3a59bf501597c [sys.catalog]: This master's current role is: LEADER
I20260806 02:51:58.255686 28297 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:51:58.255528 28289 sys_catalog.cc:455] T 00000000000000000000000000000000 P db26e8a240ca4210a0c3a59bf501597c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db26e8a240ca4210a0c3a59bf501597c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db26e8a240ca4210a0c3a59bf501597c" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 41109 } } }
I20260806 02:51:58.255861 28289 sys_catalog.cc:458] T 00000000000000000000000000000000 P db26e8a240ca4210a0c3a59bf501597c [sys.catalog]: This master's current role is: LEADER
I20260806 02:51:58.262310 28297 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:51:58.273509 28297 catalog_manager.cc:1347] Generated new cluster ID: 67c31317a82c4faa8aaac78bd9e6d88f
I20260806 02:51:58.273654 28297 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:51:58.289242 28297 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:51:58.291326 28297 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:51:58.298874 28297 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db26e8a240ca4210a0c3a59bf501597c: Generated new TSK 0
I20260806 02:51:58.299973 28297 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.211.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab.
Entry for principal kudu/127.25.211.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab.
I20260806 02:51:58.353586 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.129:0
--local_ip_for_outbound_sockets=127.25.211.129
--webserver_interface=127.25.211.129
--webserver_port=0
--tserver_master_addrs=127.25.211.190:41109
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab
--principal=kudu/127.25.211.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:35545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:51:58.697538 28322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:51:58.697867 28322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:51:58.697969 28322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:51:58.705480 28322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:51:58.705634 28322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:51:58.705739 28322 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 02:51:58.705826 28322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.129
I20260806 02:51:58.715919 28322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:35545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab
--principal=kudu/127.25.211.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.211.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.25.211.190:41109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.129
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:51:58.717327 28322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:51:58.718783 28322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:51:58.732170 28327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:58.732241 28328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:58.732213 28330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:58.733072 28322 server_base.cc:1034] running on GCE node
I20260806 02:51:58.733771 28322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:51:58.734787 28322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:51:58.736012 28322 hybrid_clock.cc:648] HybridClock initialized: now 1785984718735947 us; error 38 us; skew 500 ppm
I20260806 02:51:58.736346 28322 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:51:58 dist-test-slave-zj8q krb5kdc[28137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984718, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:51:58.739807 28322 init.cc:377] Logged in from keytab as kudu/127.25.211.129@KRBTEST.COM (short username kudu)
I20260806 02:51:58.741375 28322 webserver.cc:460] Webserver started at http://127.25.211.129:37683/ using document root <none> and password file <none>
I20260806 02:51:58.742141 28322 fs_manager.cc:359] Metadata directory not provided
I20260806 02:51:58.742312 28322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:51:58.742650 28322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:51:58.745847 28322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/ts-0/data/instance:
uuid: "d558e8d78f684854947a74cd5718a5c7"
format_stamp: "Formatted at 2026-08-06 02:51:58 on dist-test-slave-zj8q"
server_key: "a21d13df4fc82d39c2892972a4d6fce1"
server_key_iv: "c2211c24c792f788ed8b4f9e64f80976"
server_key_version: "encryptionkey@0"
I20260806 02:51:58.746801 28322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/ts-0/wal/instance:
uuid: "d558e8d78f684854947a74cd5718a5c7"
format_stamp: "Formatted at 2026-08-06 02:51:58 on dist-test-slave-zj8q"
server_key: "a21d13df4fc82d39c2892972a4d6fce1"
server_key_iv: "c2211c24c792f788ed8b4f9e64f80976"
server_key_version: "encryptionkey@0"
I20260806 02:51:58.752540 28322 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 02:51:58.756417 28340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:58.757942 28322 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:51:58.758181 28322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/ts-0/wal
uuid: "d558e8d78f684854947a74cd5718a5c7"
format_stamp: "Formatted at 2026-08-06 02:51:58 on dist-test-slave-zj8q"
server_key: "a21d13df4fc82d39c2892972a4d6fce1"
server_key_iv: "c2211c24c792f788ed8b4f9e64f80976"
server_key_version: "encryptionkey@0"
I20260806 02:51:58.758471 28322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:51:58.776289 28322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:51:58.780809 28322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:51:58.781049 28322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:51:58.783963 28322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:51:58.784078 28322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:58.784235 28322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:51:58.784299 28322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 02:51:58 dist-test-slave-zj8q krb5kdc[28137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984718, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.129@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:51:58.808990 28351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:58.784467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:42647 (local address 127.25.211.190:41109)
0806 02:51:58.785286 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:58.785292 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:58.786614 (+ 1322us) server_negotiation.cc:407] Connection header received
0806 02:51:58.788888 (+ 2274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:58.788894 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:58.788991 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:58.789088 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:58.791557 (+ 2469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:58.792553 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:58.795601 (+ 3048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:58.796118 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:58.796258 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:58.801856 (+ 5598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:58.801865 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:58.801871 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:58.801927 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:58.805274 (+ 3347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:58.806139 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:58.806144 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:58.806147 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:58.806364 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:58.807034 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:58.807038 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:58.807040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:58.807377 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:58.807496 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:58.808518 (+ 1022us) server_negotiation.cc:299] Negotiation successful
0806 02:51:58.808729 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":702,"thread_start_us":610,"threads_started":1}
I20260806 02:51:58.811163 28353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:58.785119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:41109 (local address 127.25.211.129:42647)
0806 02:51:58.786433 (+ 1314us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:58.786480 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:58.788567 (+ 2087us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:58.789417 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:58.789432 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:58.790359 (+ 927us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:58.791354 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:58.791383 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:58.792749 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:58.792756 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:58.795411 (+ 2655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:58.795432 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:58.796393 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:58.796401 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:58.796743 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:58.797922 (+ 1179us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:58.797961 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:58.801531 (+ 3570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:51:58.805445 (+ 3914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:58.805456 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:58.805477 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:58.805992 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:58.806506 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:58.806513 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:58.806517 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:58.806888 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:58.807499 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:51:58.807510 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:51:58.808083 (+ 573us) client_negotiation.cc:769] Sending connection context
0806 02:51:58.808449 (+ 366us) client_negotiation.cc:240] Negotiation successful
0806 02:51:58.809122 (+ 673us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":935,"thread_start_us":627,"threads_started":1}
I20260806 02:51:58.832938 28322 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.129:45475
I20260806 02:51:58.832953 28531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.129:45475 every 8 connection(s)
I20260806 02:51:58.834240 28481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:58.817916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:41109 (local address 127.25.211.129:55015)
0806 02:51:58.819441 (+ 1525us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:58.819464 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:58.819643 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:58.820019 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:58.820024 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:58.820686 (+ 662us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:58.821103 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:58.821119 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:58.822278 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:58.822284 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:58.824243 (+ 1959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:58.824264 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:58.825055 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:58.825063 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:58.825289 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:58.826085 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:58.826110 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:58.827386 (+ 1276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:51:58.831501 (+ 4115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:58.831508 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:58.831512 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:58.831996 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:58.832479 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:58.832488 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:58.832492 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:58.832659 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:58.833290 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:51:58.833293 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:51:58.833466 (+ 173us) client_negotiation.cc:769] Sending connection context
0806 02:51:58.833591 (+ 125us) client_negotiation.cc:240] Negotiation successful
0806 02:51:58.833869 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1339,"thread_start_us":1069,"threads_started":1}
I20260806 02:51:58.834369 28351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:58.817844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:55015 (local address 127.25.211.190:41109)
0806 02:51:58.818045 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:58.818051 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:58.819572 (+ 1521us) server_negotiation.cc:407] Connection header received
0806 02:51:58.819806 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:58.819811 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:58.819883 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:58.819975 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:58.821313 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:58.822084 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:58.824412 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:58.824817 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:58.824926 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:58.827765 (+ 2839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:58.827774 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:58.827779 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:58.827831 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:58.831342 (+ 3511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:58.832198 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:58.832204 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:58.832207 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:58.832358 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:58.832772 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:58.832778 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:58.832782 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:58.833093 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:58.833228 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:58.833753 (+ 525us) server_negotiation.cc:299] Negotiation successful
0806 02:51:58.834008 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 02:51:58.834673 28322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
I20260806 02:51:58.835597 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 28322
I20260806 02:51:58.835829 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/ts-0/wal/instance
I20260806 02:51:58.836634 26446 external_mini_cluster.cc:1428] Setting key 883739f565e20713e8a303588efcd6cb
I20260806 02:51:58.839329 28532 heartbeater.cc:347] Connected to a master server at 127.25.211.190:41109
I20260806 02:51:58.839643 28532 heartbeater.cc:464] Registering TS with master...
I20260806 02:51:58.840502 28532 heartbeater.cc:511] Master 127.25.211.190:41109 requested a full tablet report, sending...
I20260806 02:51:58.842955 28205 ts_manager.cc:194] Registered new tserver with Master: d558e8d78f684854947a74cd5718a5c7 (127.25.211.129:45475)
I20260806 02:51:58.844440 28205 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.211.129@KRBTEST.COM'} at 127.25.211.129:42647
WARNING: no policy specified for kudu/127.25.211.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab.
Entry for principal kudu/127.25.211.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab.
I20260806 02:51:58.883643 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.130:0
--local_ip_for_outbound_sockets=127.25.211.130
--webserver_interface=127.25.211.130
--webserver_port=0
--tserver_master_addrs=127.25.211.190:41109
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab
--principal=kudu/127.25.211.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:35545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:51:59.229246 28536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:51:59.229569 28536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:51:59.229702 28536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:51:59.237315 28536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:51:59.237455 28536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:51:59.237574 28536 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 02:51:59.237691 28536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.130
I20260806 02:51:59.248068 28536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:35545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab
--principal=kudu/127.25.211.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.211.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.25.211.190:41109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.130
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:51:59.249558 28536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:51:59.251046 28536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:51:59.264807 28550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:59.264905 28549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:59.264825 28552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:59.265897 28536 server_base.cc:1034] running on GCE node
I20260806 02:51:59.266644 28536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:51:59.267634 28536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:51:59.268869 28536 hybrid_clock.cc:648] HybridClock initialized: now 1785984719268799 us; error 39 us; skew 500 ppm
I20260806 02:51:59.269186 28536 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:51:59 dist-test-slave-zj8q krb5kdc[28137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984719, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:51:59.276707 28536 init.cc:377] Logged in from keytab as kudu/127.25.211.130@KRBTEST.COM (short username kudu)
I20260806 02:51:59.278368 28536 webserver.cc:460] Webserver started at http://127.25.211.130:33473/ using document root <none> and password file <none>
I20260806 02:51:59.279150 28536 fs_manager.cc:359] Metadata directory not provided
I20260806 02:51:59.279277 28536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:51:59.279619 28536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:51:59.282968 28536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/ts-1/data/instance:
uuid: "c4cedeee78a4476b802e4bad356c31a2"
format_stamp: "Formatted at 2026-08-06 02:51:59 on dist-test-slave-zj8q"
server_key: "4be87eef92e86599bf88dbbf4785de59"
server_key_iv: "4d4d0dc8c59ec73055314e4bdb97dddd"
server_key_version: "encryptionkey@0"
I20260806 02:51:59.283869 28536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/ts-1/wal/instance:
uuid: "c4cedeee78a4476b802e4bad356c31a2"
format_stamp: "Formatted at 2026-08-06 02:51:59 on dist-test-slave-zj8q"
server_key: "4be87eef92e86599bf88dbbf4785de59"
server_key_iv: "4d4d0dc8c59ec73055314e4bdb97dddd"
server_key_version: "encryptionkey@0"
I20260806 02:51:59.289856 28536 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260806 02:51:59.293853 28559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:59.295418 28536 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:51:59.295604 28536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/ts-1/wal
uuid: "c4cedeee78a4476b802e4bad356c31a2"
format_stamp: "Formatted at 2026-08-06 02:51:59 on dist-test-slave-zj8q"
server_key: "4be87eef92e86599bf88dbbf4785de59"
server_key_iv: "4d4d0dc8c59ec73055314e4bdb97dddd"
server_key_version: "encryptionkey@0"
I20260806 02:51:59.295837 28536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:51:59.311251 28536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:51:59.315910 28536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:51:59.316190 28536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:51:59.319082 28536 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:51:59.319175 28536 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:59.319305 28536 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:51:59.319373 28536 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 02:51:59 dist-test-slave-zj8q krb5kdc[28137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984719, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.130@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:51:59.344795 28351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:59.319701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:46741 (local address 127.25.211.190:41109)
0806 02:51:59.319899 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:59.319905 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:59.321308 (+ 1403us) server_negotiation.cc:407] Connection header received
0806 02:51:59.323568 (+ 2260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:59.323575 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:59.323671 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:59.323738 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:59.326384 (+ 2646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:59.327424 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:59.330512 (+ 3088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:59.331134 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:59.331278 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:59.336533 (+ 5255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:59.336551 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:59.336557 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:59.336610 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:59.340432 (+ 3822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:59.341747 (+ 1315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:59.341752 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:59.341754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:59.341929 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:59.342647 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:59.342653 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:59.342656 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:59.343053 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:59.343239 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:59.344337 (+ 1098us) server_negotiation.cc:299] Negotiation successful
0806 02:51:59.344537 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260806 02:51:59.346592 28574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:59.320228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:41109 (local address 127.25.211.130:46741)
0806 02:51:59.321117 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:59.321172 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:59.323179 (+ 2007us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:59.324294 (+ 1115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:59.324304 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:59.325188 (+ 884us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:59.326104 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:59.326131 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:59.327614 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:59.327621 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:59.330304 (+ 2683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:59.330325 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:59.331417 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:59.331426 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:59.331788 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:59.332639 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:59.332668 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:59.336160 (+ 3492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:51:59.340643 (+ 4483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:59.340656 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:59.340693 (+ 37us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:59.341577 (+ 884us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:59.342069 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:59.342075 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:59.342079 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:59.342473 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:59.343235 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:51:59.343249 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:51:59.343847 (+ 598us) client_negotiation.cc:769] Sending connection context
0806 02:51:59.344201 (+ 354us) client_negotiation.cc:240] Negotiation successful
0806 02:51:59.344862 (+ 661us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":347,"threads_started":1}
I20260806 02:51:59.362380 28536 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.130:45575
I20260806 02:51:59.362495 28743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.130:45575 every 8 connection(s)
I20260806 02:51:59.363945 28536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
I20260806 02:51:59.366349 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 28536
I20260806 02:51:59.366559 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/ts-1/wal/instance
I20260806 02:51:59.366513 28746 heartbeater.cc:347] Connected to a master server at 127.25.211.190:41109
I20260806 02:51:59.366808 28746 heartbeater.cc:464] Registering TS with master...
I20260806 02:51:59.366954 26446 external_mini_cluster.cc:1428] Setting key 61c254c5b8c24fb395a2f1956daff473
I20260806 02:51:59.367707 28746 heartbeater.cc:511] Master 127.25.211.190:41109 requested a full tablet report, sending...
I20260806 02:51:59.369423 28205 ts_manager.cc:194] Registered new tserver with Master: c4cedeee78a4476b802e4bad356c31a2 (127.25.211.130:45575)
I20260806 02:51:59.370527 28205 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.211.130@KRBTEST.COM'} at 127.25.211.130:46741
I20260806 02:51:59.382378 28714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:59.354342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:41109 (local address 127.25.211.130:39199)
0806 02:51:59.369108 (+ 14766us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:59.369131 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:59.369281 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:59.369772 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:59.369776 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:59.370283 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:59.370546 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:59.370556 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:59.371949 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:59.371955 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:59.373401 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:59.373417 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:59.374213 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:59.374220 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:59.374390 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:59.375008 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:59.375031 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:59.375985 (+ 954us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:51:59.379750 (+ 3765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:59.379755 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:59.379758 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:59.380224 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:59.380653 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:59.380657 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:59.380659 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:59.380782 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:59.381457 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:51:59.381460 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:51:59.381663 (+ 203us) client_negotiation.cc:769] Sending connection context
0806 02:51:59.381773 (+ 110us) client_negotiation.cc:240] Negotiation successful
0806 02:51:59.382002 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14591,"thread_start_us":15014,"threads_started":1}
I20260806 02:51:59.382509 28351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:59.354326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:39199 (local address 127.25.211.190:41109)
0806 02:51:59.354569 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:59.354577 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:59.369219 (+ 14642us) server_negotiation.cc:407] Connection header received
0806 02:51:59.369399 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:59.369408 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:59.369492 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:59.369589 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:59.370706 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:59.371759 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:59.373572 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:59.373998 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:59.374100 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:59.376250 (+ 2150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:59.376259 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:59.376264 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:59.376307 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:59.379596 (+ 3289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:59.380375 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:59.380382 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:59.380386 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:59.380556 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:59.380884 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:59.380888 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:59.380891 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:59.381350 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:59.381493 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:59.381910 (+ 417us) server_negotiation.cc:299] Negotiation successful
0806 02:51:59.382210 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
WARNING: no policy specified for kudu/127.25.211.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab.
Entry for principal kudu/127.25.211.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab.
I20260806 02:51:59.417081 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.131:0
--local_ip_for_outbound_sockets=127.25.211.131
--webserver_interface=127.25.211.131
--webserver_port=0
--tserver_master_addrs=127.25.211.190:41109
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab
--principal=kudu/127.25.211.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:35545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:51:59.754686 28751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:51:59.754993 28751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:51:59.755131 28751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:51:59.762498 28751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:51:59.762682 28751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:51:59.762807 28751 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 02:51:59.762918 28751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.131
I20260806 02:51:59.772789 28751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:35545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab
--principal=kudu/127.25.211.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.211.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.25.211.190:41109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.131
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:51:59.774130 28751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:51:59.775568 28751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:51:59.788945 28763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:59.788734 28764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:51:59.788856 28767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:51:59.789983 28751 server_base.cc:1034] running on GCE node
I20260806 02:51:59.790707 28751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:51:59.794090 28751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:51:59.795543 28751 hybrid_clock.cc:648] HybridClock initialized: now 1785984719795468 us; error 46 us; skew 500 ppm
I20260806 02:51:59.795907 28751 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:51:59 dist-test-slave-zj8q krb5kdc[28137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984719, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:51:59.799341 28751 init.cc:377] Logged in from keytab as kudu/127.25.211.131@KRBTEST.COM (short username kudu)
I20260806 02:51:59.801083 28751 webserver.cc:460] Webserver started at http://127.25.211.131:45613/ using document root <none> and password file <none>
I20260806 02:51:59.801853 28751 fs_manager.cc:359] Metadata directory not provided
I20260806 02:51:59.801978 28751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:51:59.802385 28751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:51:59.805644 28751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/ts-2/data/instance:
uuid: "bcda2fddf1674c178cb2988bbc77d005"
format_stamp: "Formatted at 2026-08-06 02:51:59 on dist-test-slave-zj8q"
server_key: "c317f60e4b6de122f88f9f1fac79061b"
server_key_iv: "c9af4c0292c604952e202eeea8224540"
server_key_version: "encryptionkey@0"
I20260806 02:51:59.806591 28751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/ts-2/wal/instance:
uuid: "bcda2fddf1674c178cb2988bbc77d005"
format_stamp: "Formatted at 2026-08-06 02:51:59 on dist-test-slave-zj8q"
server_key: "c317f60e4b6de122f88f9f1fac79061b"
server_key_iv: "c9af4c0292c604952e202eeea8224540"
server_key_version: "encryptionkey@0"
I20260806 02:51:59.812134 28751 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 02:51:59.816241 28776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:59.817844 28751 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 02:51:59.818037 28751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/ts-2/wal
uuid: "bcda2fddf1674c178cb2988bbc77d005"
format_stamp: "Formatted at 2026-08-06 02:51:59 on dist-test-slave-zj8q"
server_key: "c317f60e4b6de122f88f9f1fac79061b"
server_key_iv: "c9af4c0292c604952e202eeea8224540"
server_key_version: "encryptionkey@0"
I20260806 02:51:59.818320 28751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:51:59.847594 28532 heartbeater.cc:503] Master 127.25.211.190:41109 was elected leader, sending a full tablet report...
I20260806 02:51:59.848492 28751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:51:59.853113 28751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:51:59.853392 28751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:51:59.856419 28751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:51:59.856527 28751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:51:59.856674 28751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:51:59.856761 28751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 02:51:59 dist-test-slave-zj8q krb5kdc[28137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984719, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.131@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:51:59.880039 28351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:59.856914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:57951 (local address 127.25.211.190:41109)
0806 02:51:59.857129 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:59.857136 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:59.858750 (+ 1614us) server_negotiation.cc:407] Connection header received
0806 02:51:59.861190 (+ 2440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:59.861195 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:59.861285 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:59.861371 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:59.864139 (+ 2768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:59.865323 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:59.867662 (+ 2339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:59.868327 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:59.868476 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:59.872617 (+ 4141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:59.872626 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:59.872632 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:59.872678 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:59.876348 (+ 3670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:59.877231 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:59.877238 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:59.877241 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:59.877440 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:59.878019 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:59.878023 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:59.878025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:59.878378 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:59.878521 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:59.879409 (+ 888us) server_negotiation.cc:299] Negotiation successful
0806 02:51:59.879691 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 02:51:59.881150 28792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:59.857699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:41109 (local address 127.25.211.131:57951)
0806 02:51:59.858580 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:59.858630 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:59.860848 (+ 2218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:59.861699 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:59.861712 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:59.862756 (+ 1044us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:59.863895 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:59.863929 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:59.865532 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:59.865538 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:59.867497 (+ 1959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:59.867524 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:59.868577 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:59.868584 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:59.868774 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:59.869576 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:59.869602 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:59.872324 (+ 2722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:51:59.876530 (+ 4206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:59.876540 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:59.876567 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:59.877081 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:59.877592 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:59.877597 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:59.877601 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:59.877891 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:59.878516 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:51:59.878525 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:51:59.878932 (+ 407us) client_negotiation.cc:769] Sending connection context
0806 02:51:59.879217 (+ 285us) client_negotiation.cc:240] Negotiation successful
0806 02:51:59.879801 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":394,"threads_started":1}
I20260806 02:51:59.903744 28351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:59.887356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:37145 (local address 127.25.211.190:41109)
0806 02:51:59.887542 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:59.887548 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:59.890723 (+ 3175us) server_negotiation.cc:407] Connection header received
0806 02:51:59.890904 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:59.890910 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:59.891005 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:59.891101 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:59.892273 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:59.892985 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:59.894371 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:59.894963 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:59.895080 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:59.897489 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:59.897497 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:59.897502 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:59.897545 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:59.900819 (+ 3274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:59.901692 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:59.901698 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:59.901702 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:59.901882 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:59.902271 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:59.902277 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:59.902280 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:59.902644 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:59.902821 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:59.903313 (+ 492us) server_negotiation.cc:299] Negotiation successful
0806 02:51:59.903494 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 02:51:59.904102 28881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:59.887433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:41109 (local address 127.25.211.131:37145)
0806 02:51:59.890592 (+ 3159us) negotiation.cc:107] Waiting for socket to connect
0806 02:51:59.890610 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:51:59.890775 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:51:59.891158 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:51:59.891162 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:51:59.891806 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:51:59.892113 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:59.892122 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:59.893124 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:59.893128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:59.894213 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:51:59.894229 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:51:59.895180 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:51:59.895187 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:51:59.895417 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:59.896142 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:51:59.896169 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:51:59.897253 (+ 1084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:51:59.901008 (+ 3755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:59.901014 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:59.901018 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:59.901563 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:59.902001 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:51:59.902006 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:51:59.902010 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:51:59.902144 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:51:59.902795 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:51:59.902802 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:51:59.903040 (+ 238us) client_negotiation.cc:769] Sending connection context
0806 02:51:59.903216 (+ 176us) client_negotiation.cc:240] Negotiation successful
0806 02:51:59.903614 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3019,"thread_start_us":2883,"threads_started":1}
I20260806 02:51:59.913930 28751 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.131:44023
I20260806 02:51:59.913947 28946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.131:44023 every 8 connection(s)
I20260806 02:51:59.915349 28751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
I20260806 02:51:59.917649 28947 heartbeater.cc:347] Connected to a master server at 127.25.211.190:41109
I20260806 02:51:59.917865 28947 heartbeater.cc:464] Registering TS with master...
I20260806 02:51:59.918661 28947 heartbeater.cc:511] Master 127.25.211.190:41109 requested a full tablet report, sending...
I20260806 02:51:59.918741 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 28751
I20260806 02:51:59.918936 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/ts-2/wal/instance
I20260806 02:51:59.919473 26446 external_mini_cluster.cc:1428] Setting key e93ddc246147cb08d2a5b53586532c31
I20260806 02:51:59.920347 28205 ts_manager.cc:194] Registered new tserver with Master: bcda2fddf1674c178cb2988bbc77d005 (127.25.211.131:44023)
I20260806 02:51:59.921458 28205 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.211.131@KRBTEST.COM'} at 127.25.211.131:57951
I20260806 02:51:59.922832 26446 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 02:51:59 dist-test-slave-zj8q krb5kdc[28137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984719, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 02:51:59 dist-test-slave-zj8q krb5kdc[28137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984719, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:51:59.976761 28351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:51:59.962250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44774 (local address 127.25.211.190:41109)
0806 02:51:59.962479 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0806 02:51:59.962486 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:51:59.962797 (+ 311us) server_negotiation.cc:407] Connection header received
0806 02:51:59.963074 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:51:59.963078 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:51:59.963153 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:51:59.963245 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:51:59.964707 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:59.965488 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:59.966904 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:51:59.967364 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:51:59.967473 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:51:59.971515 (+ 4042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:51:59.971521 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:51:59.971525 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:51:59.971570 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:51:59.974047 (+ 2477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:59.974798 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:59.974802 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:59.974805 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:59.974940 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:51:59.975261 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:51:59.975264 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:51:59.975266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:51:59.975547 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:51:59.975803 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0806 02:51:59.976164 (+ 361us) server_negotiation.cc:299] Negotiation successful
0806 02:51:59.976415 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 02:51:59.981439 28204 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44774:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:51:59.988590 28204 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:52:00.027462 28970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:00.001098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:45475 (local address 127.0.0.1:55794)
0806 02:52:00.012318 (+ 11220us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:00.012373 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:00.012621 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:00.016280 (+ 3659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:00.016290 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:00.016357 (+ 67us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:52:00.016839 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:00.016856 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:00.019231 (+ 2375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:00.019238 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:00.024074 (+ 4836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:00.024102 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:00.026562 (+ 2460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:00.026573 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:00.026758 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:00.026797 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 02:52:00.026895 (+ 98us) client_negotiation.cc:240] Negotiation successful
0806 02:52:00.027040 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10897,"mutex_wait_us":54,"thread_start_us":24954,"threads_started":1}
I20260806 02:52:00.028198 28972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:00.001378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55794 (local address 127.25.211.129:45475)
0806 02:52:00.015488 (+ 14110us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:00.015497 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:00.015533 (+ 36us) server_negotiation.cc:407] Connection header received
0806 02:52:00.015650 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:00.015660 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:00.015967 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:00.016194 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:52:00.017096 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:00.018995 (+ 1899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:00.024344 (+ 5349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:00.026028 (+ 1684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:00.026216 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:00.026381 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:00.027725 (+ 1344us) server_negotiation.cc:299] Negotiation successful
0806 02:52:00.027907 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13926,"thread_start_us":13701,"threads_started":1}
I20260806 02:52:00.028236 28975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:00.001900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49282 (local address 127.25.211.131:44023)
0806 02:52:00.017961 (+ 16061us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:00.017970 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:00.018008 (+ 38us) server_negotiation.cc:407] Connection header received
0806 02:52:00.018109 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:00.018116 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:00.018445 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:00.018673 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:52:00.019839 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:00.021529 (+ 1690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:00.024349 (+ 2820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:00.025443 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:00.025667 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:00.025875 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:00.027696 (+ 1821us) server_negotiation.cc:299] Negotiation successful
0806 02:52:00.027904 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15869,"thread_start_us":15653,"threads_started":1}
I20260806 02:52:00.028424 28973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:00.001227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.131:44023 (local address 127.0.0.1:49282)
0806 02:52:00.013623 (+ 12396us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:00.013644 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:00.013802 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:00.019029 (+ 5227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:00.019034 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:00.019063 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:52:00.019592 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:00.019609 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:00.021717 (+ 2108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:00.021723 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:00.024122 (+ 2399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:00.024141 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:00.025687 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:00.025694 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:00.025927 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:00.026019 (+ 92us) client_negotiation.cc:769] Sending connection context
0806 02:52:00.026236 (+ 217us) client_negotiation.cc:240] Negotiation successful
0806 02:52:00.026456 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12247,"mutex_wait_us":3,"thread_start_us":10597,"threads_started":1}
I20260806 02:52:00.032914 28853 tablet_service.cc:1467] Processing CreateTablet for tablet 4c642bafad0041dfb285dfc49b47e3e5 (DEFAULT_TABLE table=test-table [id=419222f40d064106a06edba1d5288200]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:52:00.032938 28425 tablet_service.cc:1467] Processing CreateTablet for tablet 4c642bafad0041dfb285dfc49b47e3e5 (DEFAULT_TABLE table=test-table [id=419222f40d064106a06edba1d5288200]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:52:00.034502 28853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c642bafad0041dfb285dfc49b47e3e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:00.034592 28425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c642bafad0041dfb285dfc49b47e3e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:00.035157 28969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:00.001668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.130:45575 (local address 127.0.0.1:53874)
0806 02:52:00.026386 (+ 24718us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:00.026405 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:00.026546 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:00.027830 (+ 1284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:00.027835 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:00.027862 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:52:00.028224 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:00.028237 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:00.029777 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:00.029783 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:00.031854 (+ 2071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:00.031877 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:00.034358 (+ 2481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:00.034365 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:00.034535 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:00.034570 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 02:52:00.034666 (+ 96us) client_negotiation.cc:240] Negotiation successful
0806 02:52:00.034822 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24555,"thread_start_us":12536,"threads_started":1}
I20260806 02:52:00.035876 28978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:00.002132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53874 (local address 127.25.211.130:45575)
0806 02:52:00.013104 (+ 10972us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:00.013115 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:00.026521 (+ 13406us) server_negotiation.cc:407] Connection header received
0806 02:52:00.026848 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:00.026857 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:00.027160 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:00.027399 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:52:00.028450 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:00.029627 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:00.032145 (+ 2518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:00.033795 (+ 1650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:00.034008 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:00.034214 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:00.035401 (+ 1187us) server_negotiation.cc:299] Negotiation successful
0806 02:52:00.035590 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10777,"thread_start_us":11785,"threads_started":1}
I20260806 02:52:00.039732 28645 tablet_service.cc:1467] Processing CreateTablet for tablet 4c642bafad0041dfb285dfc49b47e3e5 (DEFAULT_TABLE table=test-table [id=419222f40d064106a06edba1d5288200]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:52:00.041244 28645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c642bafad0041dfb285dfc49b47e3e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:00.046360 28979 tablet_bootstrap.cc:492] T 4c642bafad0041dfb285dfc49b47e3e5 P d558e8d78f684854947a74cd5718a5c7: Bootstrap starting.
I20260806 02:52:00.050062 28979 tablet_bootstrap.cc:654] T 4c642bafad0041dfb285dfc49b47e3e5 P d558e8d78f684854947a74cd5718a5c7: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:00.051184 28979 log.cc:824] T 4c642bafad0041dfb285dfc49b47e3e5 P d558e8d78f684854947a74cd5718a5c7: Log is configured to *not* fsync() on all Append() calls
I20260806 02:52:00.052197 28980 tablet_bootstrap.cc:492] T 4c642bafad0041dfb285dfc49b47e3e5 P bcda2fddf1674c178cb2988bbc77d005: Bootstrap starting.
I20260806 02:52:00.056109 28979 tablet_bootstrap.cc:492] T 4c642bafad0041dfb285dfc49b47e3e5 P d558e8d78f684854947a74cd5718a5c7: No bootstrap required, opened a new log
I20260806 02:52:00.056502 28980 tablet_bootstrap.cc:654] T 4c642bafad0041dfb285dfc49b47e3e5 P bcda2fddf1674c178cb2988bbc77d005: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:00.056550 28979 ts_tablet_manager.cc:1397] T 4c642bafad0041dfb285dfc49b47e3e5 P d558e8d78f684854947a74cd5718a5c7: Time spent bootstrapping tablet: real 0.010s user 0.010s sys 0.000s
I20260806 02:52:00.058183 28983 tablet_bootstrap.cc:492] T 4c642bafad0041dfb285dfc49b47e3e5 P c4cedeee78a4476b802e4bad356c31a2: Bootstrap starting.
I20260806 02:52:00.058790 28980 log.cc:824] T 4c642bafad0041dfb285dfc49b47e3e5 P bcda2fddf1674c178cb2988bbc77d005: Log is configured to *not* fsync() on all Append() calls
I20260806 02:52:00.062740 28983 tablet_bootstrap.cc:654] T 4c642bafad0041dfb285dfc49b47e3e5 P c4cedeee78a4476b802e4bad356c31a2: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:00.064265 28983 log.cc:824] T 4c642bafad0041dfb285dfc49b47e3e5 P c4cedeee78a4476b802e4bad356c31a2: Log is configured to *not* fsync() on all Append() calls
I20260806 02:52:00.066089 28980 tablet_bootstrap.cc:492] T 4c642bafad0041dfb285dfc49b47e3e5 P bcda2fddf1674c178cb2988bbc77d005: No bootstrap required, opened a new log
I20260806 02:52:00.066529 28980 ts_tablet_manager.cc:1397] T 4c642bafad0041dfb285dfc49b47e3e5 P bcda2fddf1674c178cb2988bbc77d005: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.010s
I20260806 02:52:00.068914 28979 raft_consensus.cc:348] T 4c642bafad0041dfb285dfc49b47e3e5 P d558e8d78f684854947a74cd5718a5c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d558e8d78f684854947a74cd5718a5c7" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 45475 } } peers { permanent_uuid: "bcda2fddf1674c178cb2988bbc77d005" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 44023 } } peers { permanent_uuid: "c4cedeee78a4476b802e4bad356c31a2" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 45575 } }
I20260806 02:52:00.069525 28979 raft_consensus.cc:374] T 4c642bafad0041dfb285dfc49b47e3e5 P d558e8d78f684854947a74cd5718a5c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:00.069651 28979 raft_consensus.cc:729] T 4c642bafad0041dfb285dfc49b47e3e5 P d558e8d78f684854947a74cd5718a5c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d558e8d78f684854947a74cd5718a5c7, State: Initialized, Role: FOLLOWER
I20260806 02:52:00.070573 28979 consensus_queue.cc:260] T 4c642bafad0041dfb285dfc49b47e3e5 P d558e8d78f684854947a74cd5718a5c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d558e8d78f684854947a74cd5718a5c7" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 45475 } } peers { permanent_uuid: "bcda2fddf1674c178cb2988bbc77d005" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 44023 } } peers { permanent_uuid: "c4cedeee78a4476b802e4bad356c31a2" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 45575 } }
I20260806 02:52:00.071935 28983 tablet_bootstrap.cc:492] T 4c642bafad0041dfb285dfc49b47e3e5 P c4cedeee78a4476b802e4bad356c31a2: No bootstrap required, opened a new log
I20260806 02:52:00.072366 28983 ts_tablet_manager.cc:1397] T 4c642bafad0041dfb285dfc49b47e3e5 P c4cedeee78a4476b802e4bad356c31a2: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.008s
I20260806 02:52:00.072593 28979 ts_tablet_manager.cc:1428] T 4c642bafad0041dfb285dfc49b47e3e5 P d558e8d78f684854947a74cd5718a5c7: Time spent starting tablet: real 0.016s user 0.011s sys 0.003s
I20260806 02:52:00.078418 28980 raft_consensus.cc:348] T 4c642bafad0041dfb285dfc49b47e3e5 P bcda2fddf1674c178cb2988bbc77d005 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d558e8d78f684854947a74cd5718a5c7" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 45475 } } peers { permanent_uuid: "bcda2fddf1674c178cb2988bbc77d005" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 44023 } } peers { permanent_uuid: "c4cedeee78a4476b802e4bad356c31a2" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 45575 } }
I20260806 02:52:00.078935 28980 raft_consensus.cc:374] T 4c642bafad0041dfb285dfc49b47e3e5 P bcda2fddf1674c178cb2988bbc77d005 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:00.079059 28980 raft_consensus.cc:729] T 4c642bafad0041dfb285dfc49b47e3e5 P bcda2fddf1674c178cb2988bbc77d005 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcda2fddf1674c178cb2988bbc77d005, State: Initialized, Role: FOLLOWER
I20260806 02:52:00.079639 28980 consensus_queue.cc:260] T 4c642bafad0041dfb285dfc49b47e3e5 P bcda2fddf1674c178cb2988bbc77d005 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d558e8d78f684854947a74cd5718a5c7" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 45475 } } peers { permanent_uuid: "bcda2fddf1674c178cb2988bbc77d005" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 44023 } } peers { permanent_uuid: "c4cedeee78a4476b802e4bad356c31a2" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 45575 } }
I20260806 02:52:00.080854 28947 heartbeater.cc:503] Master 127.25.211.190:41109 was elected leader, sending a full tablet report...
I20260806 02:52:00.081617 28980 ts_tablet_manager.cc:1428] T 4c642bafad0041dfb285dfc49b47e3e5 P bcda2fddf1674c178cb2988bbc77d005: Time spent starting tablet: real 0.015s user 0.004s sys 0.009s
I20260806 02:52:00.081924 28983 raft_consensus.cc:348] T 4c642bafad0041dfb285dfc49b47e3e5 P c4cedeee78a4476b802e4bad356c31a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d558e8d78f684854947a74cd5718a5c7" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 45475 } } peers { permanent_uuid: "bcda2fddf1674c178cb2988bbc77d005" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 44023 } } peers { permanent_uuid: "c4cedeee78a4476b802e4bad356c31a2" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 45575 } }
I20260806 02:52:00.082439 28983 raft_consensus.cc:374] T 4c642bafad0041dfb285dfc49b47e3e5 P c4cedeee78a4476b802e4bad356c31a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:00.082559 28983 raft_consensus.cc:729] T 4c642bafad0041dfb285dfc49b47e3e5 P c4cedeee78a4476b802e4bad356c31a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4cedeee78a4476b802e4bad356c31a2, State: Initialized, Role: FOLLOWER
I20260806 02:52:00.083225 28983 consensus_queue.cc:260] T 4c642bafad0041dfb285dfc49b47e3e5 P c4cedeee78a4476b802e4bad356c31a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d558e8d78f684854947a74cd5718a5c7" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 45475 } } peers { permanent_uuid: "bcda2fddf1674c178cb2988bbc77d005" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 44023 } } peers { permanent_uuid: "c4cedeee78a4476b802e4bad356c31a2" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 45575 } }
I20260806 02:52:00.084388 28746 heartbeater.cc:503] Master 127.25.211.190:41109 was elected leader, sending a full tablet report...
I20260806 02:52:00.085093 28983 ts_tablet_manager.cc:1428] T 4c642bafad0041dfb285dfc49b47e3e5 P c4cedeee78a4476b802e4bad356c31a2: Time spent starting tablet: real 0.013s user 0.008s sys 0.003s
W20260806 02:52:00.093091 28533 tablet.cc:2369] T 4c642bafad0041dfb285dfc49b47e3e5 P d558e8d78f684854947a74cd5718a5c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:52:00.116080 28748 tablet.cc:2369] T 4c642bafad0041dfb285dfc49b47e3e5 P c4cedeee78a4476b802e4bad356c31a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:52:00.160744 28987 raft_consensus.cc:482] T 4c642bafad0041dfb285dfc49b47e3e5 P d558e8d78f684854947a74cd5718a5c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:52:00.161268 28987 raft_consensus.cc:504] T 4c642bafad0041dfb285dfc49b47e3e5 P d558e8d78f684854947a74cd5718a5c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d558e8d78f684854947a74cd5718a5c7" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 45475 } } peers { permanent_uuid: "bcda2fddf1674c178cb2988bbc77d005" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 44023 } } peers { permanent_uuid: "c4cedeee78a4476b802e4bad356c31a2" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 45575 } }
I20260806 02:52:00.163553 28987 leader_election.cc:288] T 4c642bafad0041dfb285dfc49b47e3e5 P d558e8d78f684854947a74cd5718a5c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcda2fddf1674c178cb2988bbc77d005 (127.25.211.131:44023), c4cedeee78a4476b802e4bad356c31a2 (127.25.211.130:45575)
W20260806 02:52:00.167371 28948 tablet.cc:2369] T 4c642bafad0041dfb285dfc49b47e3e5 P bcda2fddf1674c178cb2988bbc77d005: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:52:00.171250 28975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:00.163882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:55075 (local address 127.25.211.131:44023)
0806 02:52:00.164092 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:00.164098 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:00.165146 (+ 1048us) server_negotiation.cc:407] Connection header received
0806 02:52:00.165317 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:00.165322 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:00.165405 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:00.165505 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:52:00.166125 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:00.167216 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:00.169477 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:00.170321 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:00.170480 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:00.170558 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:00.170867 (+ 309us) server_negotiation.cc:299] Negotiation successful
0806 02:52:00.170995 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 02:52:00.171411 28978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:00.164408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:39763 (local address 127.25.211.130:45575)
0806 02:52:00.164605 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:00.164609 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:00.164945 (+ 336us) server_negotiation.cc:407] Connection header received
0806 02:52:00.165162 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:00.165167 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:00.165248 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:00.165334 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:52:00.166084 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:00.167161 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:00.169466 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:00.170454 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:00.170610 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:00.170696 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:00.171017 (+ 321us) server_negotiation.cc:299] Negotiation successful
0806 02:52:00.171151 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 02:52:00.171487 28996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:00.163943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.131:44023 (local address 127.25.211.129:55075)
0806 02:52:00.165017 (+ 1074us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:00.165036 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:00.165182 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:00.165548 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:00.165551 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:00.165585 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:52:00.165954 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:00.165963 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:00.167415 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:00.167421 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:00.169315 (+ 1894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:00.169329 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:00.170527 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:00.170534 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:00.170689 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:00.170742 (+ 53us) client_negotiation.cc:769] Sending connection context
0806 02:52:00.170826 (+ 84us) client_negotiation.cc:240] Negotiation successful
0806 02:52:00.170976 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":949,"mutex_wait_us":20,"thread_start_us":805,"threads_started":1}
I20260806 02:52:00.171542 28994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:00.163882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.130:45575 (local address 127.25.211.129:39763)
0806 02:52:00.164783 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:00.164808 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:00.165007 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:00.165362 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:00.165367 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:00.165400 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:52:00.165915 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:00.165927 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:00.167302 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:00.167306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:00.169253 (+ 1947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:00.169267 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:00.170689 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:00.170696 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:00.170841 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:00.170886 (+ 45us) client_negotiation.cc:769] Sending connection context
0806 02:52:00.170966 (+ 80us) client_negotiation.cc:240] Negotiation successful
0806 02:52:00.171096 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"mutex_wait_us":41,"spinlock_wait_cycles":30976,"thread_start_us":488,"threads_started":1}
I20260806 02:52:00.172501 28889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c642bafad0041dfb285dfc49b47e3e5" candidate_uuid: "d558e8d78f684854947a74cd5718a5c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcda2fddf1674c178cb2988bbc77d005" is_pre_election: true
I20260806 02:52:00.172680 28673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c642bafad0041dfb285dfc49b47e3e5" candidate_uuid: "d558e8d78f684854947a74cd5718a5c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4cedeee78a4476b802e4bad356c31a2" is_pre_election: true
I20260806 02:52:00.173009 28889 raft_consensus.cc:2454] T 4c642bafad0041dfb285dfc49b47e3e5 P bcda2fddf1674c178cb2988bbc77d005 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d558e8d78f684854947a74cd5718a5c7 in term 0.
I20260806 02:52:00.173367 28673 raft_consensus.cc:2454] T 4c642bafad0041dfb285dfc49b47e3e5 P c4cedeee78a4476b802e4bad356c31a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d558e8d78f684854947a74cd5718a5c7 in term 0.
I20260806 02:52:00.173678 28341 leader_election.cc:302] T 4c642bafad0041dfb285dfc49b47e3e5 P d558e8d78f684854947a74cd5718a5c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bcda2fddf1674c178cb2988bbc77d005, d558e8d78f684854947a74cd5718a5c7; no voters:
I20260806 02:52:00.174075 28987 raft_consensus.cc:2781] T 4c642bafad0041dfb285dfc49b47e3e5 P d558e8d78f684854947a74cd5718a5c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:52:00.174265 28987 raft_consensus.cc:482] T 4c642bafad0041dfb285dfc49b47e3e5 P d558e8d78f684854947a74cd5718a5c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:52:00.174371 28987 raft_consensus.cc:3037] T 4c642bafad0041dfb285dfc49b47e3e5 P d558e8d78f684854947a74cd5718a5c7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:00.176843 28987 raft_consensus.cc:504] T 4c642bafad0041dfb285dfc49b47e3e5 P d558e8d78f684854947a74cd5718a5c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d558e8d78f684854947a74cd5718a5c7" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 45475 } } peers { permanent_uuid: "bcda2fddf1674c178cb2988bbc77d005" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 44023 } } peers { permanent_uuid: "c4cedeee78a4476b802e4bad356c31a2" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 45575 } }
I20260806 02:52:00.177567 28987 leader_election.cc:288] T 4c642bafad0041dfb285dfc49b47e3e5 P d558e8d78f684854947a74cd5718a5c7 [CANDIDATE]: Term 1 election: Requested vote from peers bcda2fddf1674c178cb2988bbc77d005 (127.25.211.131:44023), c4cedeee78a4476b802e4bad356c31a2 (127.25.211.130:45575)
I20260806 02:52:00.178033 28889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c642bafad0041dfb285dfc49b47e3e5" candidate_uuid: "d558e8d78f684854947a74cd5718a5c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcda2fddf1674c178cb2988bbc77d005"
I20260806 02:52:00.178135 28673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c642bafad0041dfb285dfc49b47e3e5" candidate_uuid: "d558e8d78f684854947a74cd5718a5c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4cedeee78a4476b802e4bad356c31a2"
I20260806 02:52:00.178274 28889 raft_consensus.cc:3037] T 4c642bafad0041dfb285dfc49b47e3e5 P bcda2fddf1674c178cb2988bbc77d005 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:00.178408 28673 raft_consensus.cc:3037] T 4c642bafad0041dfb285dfc49b47e3e5 P c4cedeee78a4476b802e4bad356c31a2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:00.180687 28889 raft_consensus.cc:2454] T 4c642bafad0041dfb285dfc49b47e3e5 P bcda2fddf1674c178cb2988bbc77d005 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d558e8d78f684854947a74cd5718a5c7 in term 1.
I20260806 02:52:00.181180 28673 raft_consensus.cc:2454] T 4c642bafad0041dfb285dfc49b47e3e5 P c4cedeee78a4476b802e4bad356c31a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d558e8d78f684854947a74cd5718a5c7 in term 1.
I20260806 02:52:00.181236 28341 leader_election.cc:302] T 4c642bafad0041dfb285dfc49b47e3e5 P d558e8d78f684854947a74cd5718a5c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bcda2fddf1674c178cb2988bbc77d005, d558e8d78f684854947a74cd5718a5c7; no voters:
I20260806 02:52:00.181598 28987 raft_consensus.cc:2781] T 4c642bafad0041dfb285dfc49b47e3e5 P d558e8d78f684854947a74cd5718a5c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:52:00.182070 28987 raft_consensus.cc:686] T 4c642bafad0041dfb285dfc49b47e3e5 P d558e8d78f684854947a74cd5718a5c7 [term 1 LEADER]: Becoming Leader. State: Replica: d558e8d78f684854947a74cd5718a5c7, State: Running, Role: LEADER
I20260806 02:52:00.182662 28987 consensus_queue.cc:237] T 4c642bafad0041dfb285dfc49b47e3e5 P d558e8d78f684854947a74cd5718a5c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d558e8d78f684854947a74cd5718a5c7" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 45475 } } peers { permanent_uuid: "bcda2fddf1674c178cb2988bbc77d005" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 44023 } } peers { permanent_uuid: "c4cedeee78a4476b802e4bad356c31a2" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 45575 } }
I20260806 02:52:00.187953 28204 catalog_manager.cc:5495] T 4c642bafad0041dfb285dfc49b47e3e5 P d558e8d78f684854947a74cd5718a5c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d558e8d78f684854947a74cd5718a5c7 (127.25.211.129). New cstate: current_term: 1 leader_uuid: "d558e8d78f684854947a74cd5718a5c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d558e8d78f684854947a74cd5718a5c7" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 45475 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcda2fddf1674c178cb2988bbc77d005" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 44023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4cedeee78a4476b802e4bad356c31a2" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 45575 } health_report { overall_health: UNKNOWN } } }
I20260806 02:52:00.238382 28972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:00.232642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55796 (local address 127.25.211.129:45475)
0806 02:52:00.232854 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:00.232859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:00.233109 (+ 250us) server_negotiation.cc:407] Connection header received
0806 02:52:00.233437 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:00.233442 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:00.233532 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:00.233693 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:52:00.234003 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:00.234783 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:00.236220 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:00.236664 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:00.236758 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:00.237285 (+ 527us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:52:00.237848 (+ 563us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:52:00.237957 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:00.238049 (+ 92us) server_negotiation.cc:299] Negotiation successful
0806 02:52:00.238122 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 02:52:00.247212 28673 raft_consensus.cc:1261] T 4c642bafad0041dfb285dfc49b47e3e5 P c4cedeee78a4476b802e4bad356c31a2 [term 1 FOLLOWER]: Refusing update from remote peer d558e8d78f684854947a74cd5718a5c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:52:00.247212 28889 raft_consensus.cc:1261] T 4c642bafad0041dfb285dfc49b47e3e5 P bcda2fddf1674c178cb2988bbc77d005 [term 1 FOLLOWER]: Refusing update from remote peer d558e8d78f684854947a74cd5718a5c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:52:00.248171 28998 consensus_queue.cc:1035] T 4c642bafad0041dfb285dfc49b47e3e5 P d558e8d78f684854947a74cd5718a5c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcda2fddf1674c178cb2988bbc77d005" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 44023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:52:00.248638 28987 consensus_queue.cc:1035] T 4c642bafad0041dfb285dfc49b47e3e5 P d558e8d78f684854947a74cd5718a5c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4cedeee78a4476b802e4bad356c31a2" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 45575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:52:00.261466 29001 mvcc.cc:205] Tried to move back new op lower bound from 7315393414122553344 to 7315393413872476160. Current Snapshot: MvccSnapshot[applied={T|T < 7315393414122553344}]
I20260806 02:52:00.289491 28204 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:52:00.289883 28204 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:52:00.294245 28204 catalog_manager.cc:5782] T 00000000000000000000000000000000 P db26e8a240ca4210a0c3a59bf501597c: Sending DeleteTablet for 3 replicas of tablet 4c642bafad0041dfb285dfc49b47e3e5
I20260806 02:52:00.295893 28425 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c642bafad0041dfb285dfc49b47e3e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:52:00 UTC) from {username='kudu', principal='kudu/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:55794
I20260806 02:52:00.296746 28645 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c642bafad0041dfb285dfc49b47e3e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:52:00 UTC) from {username='kudu', principal='kudu/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:53874
I20260806 02:52:00.298133 28204 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44774:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260806 02:52:00.298460 28853 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c642bafad0041dfb285dfc49b47e3e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:52:00 UTC) from {username='kudu', principal='kudu/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:49282
W20260806 02:52:00.299111 28204 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:52:00.309056 28644 tablet_service.cc:1467] Processing CreateTablet for tablet d4beecde58764f6695a2e097020d59ab (DEFAULT_TABLE table=test-table [id=750ed535d2dc49faaa329bc12a54ff69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:52:00.309326 28852 tablet_service.cc:1467] Processing CreateTablet for tablet d4beecde58764f6695a2e097020d59ab (DEFAULT_TABLE table=test-table [id=750ed535d2dc49faaa329bc12a54ff69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:52:00.309880 28852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4beecde58764f6695a2e097020d59ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:00.309891 28644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4beecde58764f6695a2e097020d59ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:00.317840 28980 tablet_bootstrap.cc:492] T d4beecde58764f6695a2e097020d59ab P bcda2fddf1674c178cb2988bbc77d005: Bootstrap starting.
I20260806 02:52:00.317845 28983 tablet_bootstrap.cc:492] T d4beecde58764f6695a2e097020d59ab P c4cedeee78a4476b802e4bad356c31a2: Bootstrap starting.
I20260806 02:52:00.318341 29019 tablet_replica.cc:330] T 4c642bafad0041dfb285dfc49b47e3e5 P c4cedeee78a4476b802e4bad356c31a2: stopping tablet replica
I20260806 02:52:00.318254 28424 tablet_service.cc:1467] Processing CreateTablet for tablet d4beecde58764f6695a2e097020d59ab (DEFAULT_TABLE table=test-table [id=750ed535d2dc49faaa329bc12a54ff69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:52:00.318938 28424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4beecde58764f6695a2e097020d59ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:00.319746 28980 tablet_bootstrap.cc:654] T d4beecde58764f6695a2e097020d59ab P bcda2fddf1674c178cb2988bbc77d005: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:00.320721 28983 tablet_bootstrap.cc:654] T d4beecde58764f6695a2e097020d59ab P c4cedeee78a4476b802e4bad356c31a2: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:00.318943 29019 raft_consensus.cc:2229] T 4c642bafad0041dfb285dfc49b47e3e5 P c4cedeee78a4476b802e4bad356c31a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:52:00.327209 28979 tablet_bootstrap.cc:492] T d4beecde58764f6695a2e097020d59ab P d558e8d78f684854947a74cd5718a5c7: Bootstrap starting.
I20260806 02:52:00.328076 29019 raft_consensus.cc:2258] T 4c642bafad0041dfb285dfc49b47e3e5 P c4cedeee78a4476b802e4bad356c31a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:52:00.329551 28979 tablet_bootstrap.cc:654] T d4beecde58764f6695a2e097020d59ab P d558e8d78f684854947a74cd5718a5c7: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:00.331521 28980 tablet_bootstrap.cc:492] T d4beecde58764f6695a2e097020d59ab P bcda2fddf1674c178cb2988bbc77d005: No bootstrap required, opened a new log
I20260806 02:52:00.331786 28980 ts_tablet_manager.cc:1397] T d4beecde58764f6695a2e097020d59ab P bcda2fddf1674c178cb2988bbc77d005: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.003s
I20260806 02:52:00.332779 28980 raft_consensus.cc:348] T d4beecde58764f6695a2e097020d59ab P bcda2fddf1674c178cb2988bbc77d005 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d558e8d78f684854947a74cd5718a5c7" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 45475 } } peers { permanent_uuid: "c4cedeee78a4476b802e4bad356c31a2" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 45575 } } peers { permanent_uuid: "bcda2fddf1674c178cb2988bbc77d005" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 44023 } }
I20260806 02:52:00.333102 28980 raft_consensus.cc:374] T d4beecde58764f6695a2e097020d59ab P bcda2fddf1674c178cb2988bbc77d005 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:00.333258 28980 raft_consensus.cc:729] T d4beecde58764f6695a2e097020d59ab P bcda2fddf1674c178cb2988bbc77d005 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcda2fddf1674c178cb2988bbc77d005, State: Initialized, Role: FOLLOWER
I20260806 02:52:00.333457 29019 ts_tablet_manager.cc:1905] T 4c642bafad0041dfb285dfc49b47e3e5 P c4cedeee78a4476b802e4bad356c31a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:52:00.333700 28980 consensus_queue.cc:260] T d4beecde58764f6695a2e097020d59ab P bcda2fddf1674c178cb2988bbc77d005 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d558e8d78f684854947a74cd5718a5c7" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 45475 } } peers { permanent_uuid: "c4cedeee78a4476b802e4bad356c31a2" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 45575 } } peers { permanent_uuid: "bcda2fddf1674c178cb2988bbc77d005" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 44023 } }
I20260806 02:52:00.337895 28980 ts_tablet_manager.cc:1428] T d4beecde58764f6695a2e097020d59ab P bcda2fddf1674c178cb2988bbc77d005: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20260806 02:52:00.344209 28979 tablet_bootstrap.cc:492] T d4beecde58764f6695a2e097020d59ab P d558e8d78f684854947a74cd5718a5c7: No bootstrap required, opened a new log
I20260806 02:52:00.344501 28979 ts_tablet_manager.cc:1397] T d4beecde58764f6695a2e097020d59ab P d558e8d78f684854947a74cd5718a5c7: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.005s
I20260806 02:52:00.345651 28979 raft_consensus.cc:348] T d4beecde58764f6695a2e097020d59ab P d558e8d78f684854947a74cd5718a5c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d558e8d78f684854947a74cd5718a5c7" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 45475 } } peers { permanent_uuid: "c4cedeee78a4476b802e4bad356c31a2" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 45575 } } peers { permanent_uuid: "bcda2fddf1674c178cb2988bbc77d005" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 44023 } }
I20260806 02:52:00.345984 28979 raft_consensus.cc:374] T d4beecde58764f6695a2e097020d59ab P d558e8d78f684854947a74cd5718a5c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:00.346316 29019 ts_tablet_manager.cc:1918] T 4c642bafad0041dfb285dfc49b47e3e5 P c4cedeee78a4476b802e4bad356c31a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:52:00.346419 29020 tablet_replica.cc:330] T 4c642bafad0041dfb285dfc49b47e3e5 P bcda2fddf1674c178cb2988bbc77d005: stopping tablet replica
I20260806 02:52:00.346526 28979 raft_consensus.cc:729] T d4beecde58764f6695a2e097020d59ab P d558e8d78f684854947a74cd5718a5c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d558e8d78f684854947a74cd5718a5c7, State: Initialized, Role: FOLLOWER
I20260806 02:52:00.346890 29018 tablet_replica.cc:330] T 4c642bafad0041dfb285dfc49b47e3e5 P d558e8d78f684854947a74cd5718a5c7: stopping tablet replica
I20260806 02:52:00.347040 29020 raft_consensus.cc:2229] T 4c642bafad0041dfb285dfc49b47e3e5 P bcda2fddf1674c178cb2988bbc77d005 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:52:00.347038 28979 consensus_queue.cc:260] T d4beecde58764f6695a2e097020d59ab P d558e8d78f684854947a74cd5718a5c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d558e8d78f684854947a74cd5718a5c7" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 45475 } } peers { permanent_uuid: "c4cedeee78a4476b802e4bad356c31a2" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 45575 } } peers { permanent_uuid: "bcda2fddf1674c178cb2988bbc77d005" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 44023 } }
I20260806 02:52:00.347700 29020 raft_consensus.cc:2258] T 4c642bafad0041dfb285dfc49b47e3e5 P bcda2fddf1674c178cb2988bbc77d005 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:52:00.347687 29019 log.cc:1196] T 4c642bafad0041dfb285dfc49b47e3e5 P c4cedeee78a4476b802e4bad356c31a2: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/ts-1/wal/wals/4c642bafad0041dfb285dfc49b47e3e5
I20260806 02:52:00.347774 28979 ts_tablet_manager.cc:1428] T d4beecde58764f6695a2e097020d59ab P d558e8d78f684854947a74cd5718a5c7: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260806 02:52:00.346333 28983 tablet_bootstrap.cc:492] T d4beecde58764f6695a2e097020d59ab P c4cedeee78a4476b802e4bad356c31a2: No bootstrap required, opened a new log
I20260806 02:52:00.347294 29018 raft_consensus.cc:2229] T 4c642bafad0041dfb285dfc49b47e3e5 P d558e8d78f684854947a74cd5718a5c7 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:52:00.348393 29019 ts_tablet_manager.cc:1939] T 4c642bafad0041dfb285dfc49b47e3e5 P c4cedeee78a4476b802e4bad356c31a2: Deleting consensus metadata
I20260806 02:52:00.348500 28983 ts_tablet_manager.cc:1397] T d4beecde58764f6695a2e097020d59ab P c4cedeee78a4476b802e4bad356c31a2: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.009s
I20260806 02:52:00.350056 28983 raft_consensus.cc:348] T d4beecde58764f6695a2e097020d59ab P c4cedeee78a4476b802e4bad356c31a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d558e8d78f684854947a74cd5718a5c7" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 45475 } } peers { permanent_uuid: "c4cedeee78a4476b802e4bad356c31a2" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 45575 } } peers { permanent_uuid: "bcda2fddf1674c178cb2988bbc77d005" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 44023 } }
I20260806 02:52:00.350476 28983 raft_consensus.cc:374] T d4beecde58764f6695a2e097020d59ab P c4cedeee78a4476b802e4bad356c31a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:00.350633 28983 raft_consensus.cc:729] T d4beecde58764f6695a2e097020d59ab P c4cedeee78a4476b802e4bad356c31a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4cedeee78a4476b802e4bad356c31a2, State: Initialized, Role: FOLLOWER
I20260806 02:52:00.351080 28983 consensus_queue.cc:260] T d4beecde58764f6695a2e097020d59ab P c4cedeee78a4476b802e4bad356c31a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d558e8d78f684854947a74cd5718a5c7" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 45475 } } peers { permanent_uuid: "c4cedeee78a4476b802e4bad356c31a2" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 45575 } } peers { permanent_uuid: "bcda2fddf1674c178cb2988bbc77d005" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 44023 } }
I20260806 02:52:00.351356 29018 raft_consensus.cc:2258] T 4c642bafad0041dfb285dfc49b47e3e5 P d558e8d78f684854947a74cd5718a5c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:52:00.352986 29020 ts_tablet_manager.cc:1905] T 4c642bafad0041dfb285dfc49b47e3e5 P bcda2fddf1674c178cb2988bbc77d005: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:52:00.353307 28189 catalog_manager.cc:4841] TS c4cedeee78a4476b802e4bad356c31a2 (127.25.211.130:45575): tablet 4c642bafad0041dfb285dfc49b47e3e5 (table test-table [id=419222f40d064106a06edba1d5288200]) successfully deleted
I20260806 02:52:00.353626 29018 ts_tablet_manager.cc:1905] T 4c642bafad0041dfb285dfc49b47e3e5 P d558e8d78f684854947a74cd5718a5c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:52:00.354998 28983 ts_tablet_manager.cc:1428] T d4beecde58764f6695a2e097020d59ab P c4cedeee78a4476b802e4bad356c31a2: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260806 02:52:00.361565 29018 ts_tablet_manager.cc:1918] T 4c642bafad0041dfb285dfc49b47e3e5 P d558e8d78f684854947a74cd5718a5c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:52:00.361552 29020 ts_tablet_manager.cc:1918] T 4c642bafad0041dfb285dfc49b47e3e5 P bcda2fddf1674c178cb2988bbc77d005: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:52:00.361800 29020 log.cc:1196] T 4c642bafad0041dfb285dfc49b47e3e5 P bcda2fddf1674c178cb2988bbc77d005: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/ts-2/wal/wals/4c642bafad0041dfb285dfc49b47e3e5
I20260806 02:52:00.361802 29018 log.cc:1196] T 4c642bafad0041dfb285dfc49b47e3e5 P d558e8d78f684854947a74cd5718a5c7: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/ts-0/wal/wals/4c642bafad0041dfb285dfc49b47e3e5
I20260806 02:52:00.362521 29018 ts_tablet_manager.cc:1939] T 4c642bafad0041dfb285dfc49b47e3e5 P d558e8d78f684854947a74cd5718a5c7: Deleting consensus metadata
I20260806 02:52:00.362703 29020 ts_tablet_manager.cc:1939] T 4c642bafad0041dfb285dfc49b47e3e5 P bcda2fddf1674c178cb2988bbc77d005: Deleting consensus metadata
I20260806 02:52:00.364667 28190 catalog_manager.cc:4841] TS d558e8d78f684854947a74cd5718a5c7 (127.25.211.129:45475): tablet 4c642bafad0041dfb285dfc49b47e3e5 (table test-table [id=419222f40d064106a06edba1d5288200]) successfully deleted
I20260806 02:52:00.365015 28188 catalog_manager.cc:4841] TS bcda2fddf1674c178cb2988bbc77d005 (127.25.211.131:44023): tablet 4c642bafad0041dfb285dfc49b47e3e5 (table test-table [id=419222f40d064106a06edba1d5288200]) successfully deleted
I20260806 02:52:00.456696 28992 raft_consensus.cc:482] T d4beecde58764f6695a2e097020d59ab P c4cedeee78a4476b802e4bad356c31a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:52:00.457156 28992 raft_consensus.cc:504] T d4beecde58764f6695a2e097020d59ab P c4cedeee78a4476b802e4bad356c31a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d558e8d78f684854947a74cd5718a5c7" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 45475 } } peers { permanent_uuid: "c4cedeee78a4476b802e4bad356c31a2" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 45575 } } peers { permanent_uuid: "bcda2fddf1674c178cb2988bbc77d005" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 44023 } }
I20260806 02:52:00.459090 28992 leader_election.cc:288] T d4beecde58764f6695a2e097020d59ab P c4cedeee78a4476b802e4bad356c31a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d558e8d78f684854947a74cd5718a5c7 (127.25.211.129:45475), bcda2fddf1674c178cb2988bbc77d005 (127.25.211.131:44023)
I20260806 02:52:00.468570 28972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:00.460042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:38423 (local address 127.25.211.129:45475)
0806 02:52:00.460245 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:00.460252 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:00.461048 (+ 796us) server_negotiation.cc:407] Connection header received
0806 02:52:00.461288 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:00.461294 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:00.461402 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:00.461476 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:52:00.462698 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:00.463955 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:00.466507 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:00.467447 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:00.467637 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:00.467718 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:00.468058 (+ 340us) server_negotiation.cc:299] Negotiation successful
0806 02:52:00.468214 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 02:52:00.468750 29030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:00.459614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:45475 (local address 127.25.211.130:38423)
0806 02:52:00.460919 (+ 1305us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:00.460936 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:00.461093 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:00.461734 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:00.461739 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:00.461788 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:52:00.462523 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:00.462543 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:00.464112 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:00.464118 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:00.466328 (+ 2210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:00.466344 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:00.467683 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:00.467689 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:00.467871 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:00.467908 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 02:52:00.468012 (+ 104us) client_negotiation.cc:240] Negotiation successful
0806 02:52:00.468172 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1184,"thread_start_us":994,"threads_started":1}
I20260806 02:52:00.470105 28975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:00.459608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:35419 (local address 127.25.211.131:44023)
0806 02:52:00.459828 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:00.459835 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:00.462560 (+ 2725us) server_negotiation.cc:407] Connection header received
0806 02:52:00.462824 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:00.462830 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:00.462898 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:00.462975 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:52:00.463603 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:00.464503 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:00.467477 (+ 2974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:00.468549 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:00.468703 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:00.468772 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:00.469691 (+ 919us) server_negotiation.cc:299] Negotiation successful
0806 02:52:00.469844 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 02:52:00.470337 29033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:00.459926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.131:44023 (local address 127.25.211.130:35419)
0806 02:52:00.462409 (+ 2483us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:00.462430 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:00.462625 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:00.463024 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:00.463029 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:00.463055 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:52:00.463434 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:00.463446 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:00.464668 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:00.464674 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:00.467201 (+ 2527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:00.467219 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:00.469111 (+ 1892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:00.469117 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:00.469494 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:00.469528 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 02:52:00.469627 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 02:52:00.469763 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2255,"thread_start_us":1978,"threads_started":1}
I20260806 02:52:00.470238 28458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4beecde58764f6695a2e097020d59ab" candidate_uuid: "c4cedeee78a4476b802e4bad356c31a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d558e8d78f684854947a74cd5718a5c7" is_pre_election: true
I20260806 02:52:00.470971 28458 raft_consensus.cc:2454] T d4beecde58764f6695a2e097020d59ab P d558e8d78f684854947a74cd5718a5c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4cedeee78a4476b802e4bad356c31a2 in term 0.
I20260806 02:52:00.471225 28889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4beecde58764f6695a2e097020d59ab" candidate_uuid: "c4cedeee78a4476b802e4bad356c31a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcda2fddf1674c178cb2988bbc77d005" is_pre_election: true
I20260806 02:52:00.471546 28889 raft_consensus.cc:2454] T d4beecde58764f6695a2e097020d59ab P bcda2fddf1674c178cb2988bbc77d005 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4cedeee78a4476b802e4bad356c31a2 in term 0.
I20260806 02:52:00.471998 28565 leader_election.cc:302] T d4beecde58764f6695a2e097020d59ab P c4cedeee78a4476b802e4bad356c31a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4cedeee78a4476b802e4bad356c31a2, d558e8d78f684854947a74cd5718a5c7; no voters:
I20260806 02:52:00.472589 28992 raft_consensus.cc:2781] T d4beecde58764f6695a2e097020d59ab P c4cedeee78a4476b802e4bad356c31a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:52:00.472738 28992 raft_consensus.cc:482] T d4beecde58764f6695a2e097020d59ab P c4cedeee78a4476b802e4bad356c31a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:52:00.472838 28992 raft_consensus.cc:3037] T d4beecde58764f6695a2e097020d59ab P c4cedeee78a4476b802e4bad356c31a2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:00.475668 28992 raft_consensus.cc:504] T d4beecde58764f6695a2e097020d59ab P c4cedeee78a4476b802e4bad356c31a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d558e8d78f684854947a74cd5718a5c7" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 45475 } } peers { permanent_uuid: "c4cedeee78a4476b802e4bad356c31a2" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 45575 } } peers { permanent_uuid: "bcda2fddf1674c178cb2988bbc77d005" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 44023 } }
I20260806 02:52:00.476294 28992 leader_election.cc:288] T d4beecde58764f6695a2e097020d59ab P c4cedeee78a4476b802e4bad356c31a2 [CANDIDATE]: Term 1 election: Requested vote from peers d558e8d78f684854947a74cd5718a5c7 (127.25.211.129:45475), bcda2fddf1674c178cb2988bbc77d005 (127.25.211.131:44023)
I20260806 02:52:00.476948 28458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4beecde58764f6695a2e097020d59ab" candidate_uuid: "c4cedeee78a4476b802e4bad356c31a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d558e8d78f684854947a74cd5718a5c7"
I20260806 02:52:00.477000 28889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4beecde58764f6695a2e097020d59ab" candidate_uuid: "c4cedeee78a4476b802e4bad356c31a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcda2fddf1674c178cb2988bbc77d005"
I20260806 02:52:00.477188 28458 raft_consensus.cc:3037] T d4beecde58764f6695a2e097020d59ab P d558e8d78f684854947a74cd5718a5c7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:00.477370 28889 raft_consensus.cc:3037] T d4beecde58764f6695a2e097020d59ab P bcda2fddf1674c178cb2988bbc77d005 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:00.479835 28889 raft_consensus.cc:2454] T d4beecde58764f6695a2e097020d59ab P bcda2fddf1674c178cb2988bbc77d005 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4cedeee78a4476b802e4bad356c31a2 in term 1.
I20260806 02:52:00.480312 28563 leader_election.cc:302] T d4beecde58764f6695a2e097020d59ab P c4cedeee78a4476b802e4bad356c31a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bcda2fddf1674c178cb2988bbc77d005, c4cedeee78a4476b802e4bad356c31a2; no voters:
I20260806 02:52:00.480604 28992 raft_consensus.cc:2781] T d4beecde58764f6695a2e097020d59ab P c4cedeee78a4476b802e4bad356c31a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:52:00.480752 28458 raft_consensus.cc:2454] T d4beecde58764f6695a2e097020d59ab P d558e8d78f684854947a74cd5718a5c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4cedeee78a4476b802e4bad356c31a2 in term 1.
I20260806 02:52:00.481482 28992 raft_consensus.cc:686] T d4beecde58764f6695a2e097020d59ab P c4cedeee78a4476b802e4bad356c31a2 [term 1 LEADER]: Becoming Leader. State: Replica: c4cedeee78a4476b802e4bad356c31a2, State: Running, Role: LEADER
I20260806 02:52:00.481932 28992 consensus_queue.cc:237] T d4beecde58764f6695a2e097020d59ab P c4cedeee78a4476b802e4bad356c31a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d558e8d78f684854947a74cd5718a5c7" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 45475 } } peers { permanent_uuid: "c4cedeee78a4476b802e4bad356c31a2" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 45575 } } peers { permanent_uuid: "bcda2fddf1674c178cb2988bbc77d005" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 44023 } }
I20260806 02:52:00.485846 28205 catalog_manager.cc:5495] T d4beecde58764f6695a2e097020d59ab P c4cedeee78a4476b802e4bad356c31a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4cedeee78a4476b802e4bad356c31a2 (127.25.211.130). New cstate: current_term: 1 leader_uuid: "c4cedeee78a4476b802e4bad356c31a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d558e8d78f684854947a74cd5718a5c7" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 45475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4cedeee78a4476b802e4bad356c31a2" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 45575 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcda2fddf1674c178cb2988bbc77d005" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 44023 } health_report { overall_health: UNKNOWN } } }
Aug 06 02:52:00 dist-test-slave-zj8q krb5kdc[28137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984718, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.190@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:52:00.559458 29044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:00.544927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:41109 (local address 127.0.0.1:44778)
0806 02:52:00.545455 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:00.545480 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:00.545678 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:00.546065 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:00.546071 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:00.546815 (+ 744us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:00.547009 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:00.547019 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:00.548270 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:00.548274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:00.549406 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:00.549419 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:00.550263 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:00.550269 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:00.550427 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:00.551172 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:52:00.551189 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:00.553534 (+ 2345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:00.556789 (+ 3255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:00.556805 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:00.556814 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:00.557258 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:00.557605 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:00.557610 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:00.557614 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:00.557738 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:00.558316 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:00.558326 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:00.558481 (+ 155us) client_negotiation.cc:769] Sending connection context
0806 02:52:00.558671 (+ 190us) client_negotiation.cc:240] Negotiation successful
0806 02:52:00.558978 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":261,"threads_started":1}
I20260806 02:52:00.559724 29042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:00.544943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44778 (local address 127.25.211.190:41109)
0806 02:52:00.545809 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:00.545813 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:00.545826 (+ 13us) server_negotiation.cc:407] Connection header received
0806 02:52:00.545868 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:00.545871 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:00.545924 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:00.546013 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:00.547173 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:00.548124 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:00.549536 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:00.550079 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:00.550208 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:00.553737 (+ 3529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:00.553742 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:00.553745 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:00.553781 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:00.556645 (+ 2864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:00.557369 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:00.557375 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:00.557377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:00.557499 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:00.557830 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:00.557835 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:00.557838 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:00.558121 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:00.558286 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:00.558805 (+ 519us) server_negotiation.cc:299] Negotiation successful
0806 02:52:00.559134 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":101,"server-negotiator.queue_time_us":750,"thread_start_us":610,"threads_started":1}
I20260806 02:52:00.566365 28205 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:44778:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 02:52:00.567603 28205 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:52:00.575075 28425 tablet_service.cc:1467] Processing CreateTablet for tablet 2fe023dc39e643de8c07ad4d2c7e95e7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=52a4e13260ad4bac8df3589f29f5e13c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 02:52:00.575495 28853 tablet_service.cc:1467] Processing CreateTablet for tablet 2fe023dc39e643de8c07ad4d2c7e95e7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=52a4e13260ad4bac8df3589f29f5e13c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 02:52:00.575923 28425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fe023dc39e643de8c07ad4d2c7e95e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:00.576124 28853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fe023dc39e643de8c07ad4d2c7e95e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:00.579406 28645 tablet_service.cc:1467] Processing CreateTablet for tablet 2fe023dc39e643de8c07ad4d2c7e95e7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=52a4e13260ad4bac8df3589f29f5e13c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 02:52:00.580042 28645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fe023dc39e643de8c07ad4d2c7e95e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:00.583971 28980 tablet_bootstrap.cc:492] T 2fe023dc39e643de8c07ad4d2c7e95e7 P bcda2fddf1674c178cb2988bbc77d005: Bootstrap starting.
I20260806 02:52:00.584955 28979 tablet_bootstrap.cc:492] T 2fe023dc39e643de8c07ad4d2c7e95e7 P d558e8d78f684854947a74cd5718a5c7: Bootstrap starting.
I20260806 02:52:00.586104 28980 tablet_bootstrap.cc:654] T 2fe023dc39e643de8c07ad4d2c7e95e7 P bcda2fddf1674c178cb2988bbc77d005: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:00.587484 28979 tablet_bootstrap.cc:654] T 2fe023dc39e643de8c07ad4d2c7e95e7 P d558e8d78f684854947a74cd5718a5c7: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:00.591290 28983 tablet_bootstrap.cc:492] T 2fe023dc39e643de8c07ad4d2c7e95e7 P c4cedeee78a4476b802e4bad356c31a2: Bootstrap starting.
I20260806 02:52:00.593158 28983 tablet_bootstrap.cc:654] T 2fe023dc39e643de8c07ad4d2c7e95e7 P c4cedeee78a4476b802e4bad356c31a2: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:00.601454 28979 tablet_bootstrap.cc:492] T 2fe023dc39e643de8c07ad4d2c7e95e7 P d558e8d78f684854947a74cd5718a5c7: No bootstrap required, opened a new log
I20260806 02:52:00.601807 28979 ts_tablet_manager.cc:1397] T 2fe023dc39e643de8c07ad4d2c7e95e7 P d558e8d78f684854947a74cd5718a5c7: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.010s
I20260806 02:52:00.602856 28979 raft_consensus.cc:348] T 2fe023dc39e643de8c07ad4d2c7e95e7 P d558e8d78f684854947a74cd5718a5c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d558e8d78f684854947a74cd5718a5c7" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 45475 } } peers { permanent_uuid: "c4cedeee78a4476b802e4bad356c31a2" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 45575 } } peers { permanent_uuid: "bcda2fddf1674c178cb2988bbc77d005" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 44023 } }
I20260806 02:52:00.603098 28980 tablet_bootstrap.cc:492] T 2fe023dc39e643de8c07ad4d2c7e95e7 P bcda2fddf1674c178cb2988bbc77d005: No bootstrap required, opened a new log
I20260806 02:52:00.603271 28979 raft_consensus.cc:374] T 2fe023dc39e643de8c07ad4d2c7e95e7 P d558e8d78f684854947a74cd5718a5c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:00.603304 28983 tablet_bootstrap.cc:492] T 2fe023dc39e643de8c07ad4d2c7e95e7 P c4cedeee78a4476b802e4bad356c31a2: No bootstrap required, opened a new log
I20260806 02:52:00.603325 28980 ts_tablet_manager.cc:1397] T 2fe023dc39e643de8c07ad4d2c7e95e7 P bcda2fddf1674c178cb2988bbc77d005: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.015s
I20260806 02:52:00.603466 28983 ts_tablet_manager.cc:1397] T 2fe023dc39e643de8c07ad4d2c7e95e7 P c4cedeee78a4476b802e4bad356c31a2: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20260806 02:52:00.603466 28979 raft_consensus.cc:729] T 2fe023dc39e643de8c07ad4d2c7e95e7 P d558e8d78f684854947a74cd5718a5c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d558e8d78f684854947a74cd5718a5c7, State: Initialized, Role: FOLLOWER
I20260806 02:52:00.603962 28979 consensus_queue.cc:260] T 2fe023dc39e643de8c07ad4d2c7e95e7 P d558e8d78f684854947a74cd5718a5c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d558e8d78f684854947a74cd5718a5c7" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 45475 } } peers { permanent_uuid: "c4cedeee78a4476b802e4bad356c31a2" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 45575 } } peers { permanent_uuid: "bcda2fddf1674c178cb2988bbc77d005" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 44023 } }
I20260806 02:52:00.604143 28980 raft_consensus.cc:348] T 2fe023dc39e643de8c07ad4d2c7e95e7 P bcda2fddf1674c178cb2988bbc77d005 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d558e8d78f684854947a74cd5718a5c7" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 45475 } } peers { permanent_uuid: "c4cedeee78a4476b802e4bad356c31a2" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 45575 } } peers { permanent_uuid: "bcda2fddf1674c178cb2988bbc77d005" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 44023 } }
I20260806 02:52:00.604244 28983 raft_consensus.cc:348] T 2fe023dc39e643de8c07ad4d2c7e95e7 P c4cedeee78a4476b802e4bad356c31a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d558e8d78f684854947a74cd5718a5c7" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 45475 } } peers { permanent_uuid: "c4cedeee78a4476b802e4bad356c31a2" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 45575 } } peers { permanent_uuid: "bcda2fddf1674c178cb2988bbc77d005" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 44023 } }
I20260806 02:52:00.604455 28980 raft_consensus.cc:374] T 2fe023dc39e643de8c07ad4d2c7e95e7 P bcda2fddf1674c178cb2988bbc77d005 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:00.604496 28983 raft_consensus.cc:374] T 2fe023dc39e643de8c07ad4d2c7e95e7 P c4cedeee78a4476b802e4bad356c31a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:00.604548 28980 raft_consensus.cc:729] T 2fe023dc39e643de8c07ad4d2c7e95e7 P bcda2fddf1674c178cb2988bbc77d005 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcda2fddf1674c178cb2988bbc77d005, State: Initialized, Role: FOLLOWER
I20260806 02:52:00.604573 28983 raft_consensus.cc:729] T 2fe023dc39e643de8c07ad4d2c7e95e7 P c4cedeee78a4476b802e4bad356c31a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4cedeee78a4476b802e4bad356c31a2, State: Initialized, Role: FOLLOWER
I20260806 02:52:00.604836 28980 consensus_queue.cc:260] T 2fe023dc39e643de8c07ad4d2c7e95e7 P bcda2fddf1674c178cb2988bbc77d005 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d558e8d78f684854947a74cd5718a5c7" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 45475 } } peers { permanent_uuid: "c4cedeee78a4476b802e4bad356c31a2" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 45575 } } peers { permanent_uuid: "bcda2fddf1674c178cb2988bbc77d005" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 44023 } }
I20260806 02:52:00.604727 29011 tablet_replica.cc:439] T 2fe023dc39e643de8c07ad4d2c7e95e7 P d558e8d78f684854947a74cd5718a5c7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d558e8d78f684854947a74cd5718a5c7" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 45475 } } peers { permanent_uuid: "c4cedeee78a4476b802e4bad356c31a2" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 45575 } } peers { permanent_uuid: "bcda2fddf1674c178cb2988bbc77d005" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 44023 } } }
I20260806 02:52:00.605116 29011 tablet_replica.cc:442] T 2fe023dc39e643de8c07ad4d2c7e95e7 P d558e8d78f684854947a74cd5718a5c7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:52:00.604985 28983 consensus_queue.cc:260] T 2fe023dc39e643de8c07ad4d2c7e95e7 P c4cedeee78a4476b802e4bad356c31a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d558e8d78f684854947a74cd5718a5c7" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 45475 } } peers { permanent_uuid: "c4cedeee78a4476b802e4bad356c31a2" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 45575 } } peers { permanent_uuid: "bcda2fddf1674c178cb2988bbc77d005" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 44023 } }
I20260806 02:52:00.605293 28988 tablet_replica.cc:439] T 2fe023dc39e643de8c07ad4d2c7e95e7 P bcda2fddf1674c178cb2988bbc77d005: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d558e8d78f684854947a74cd5718a5c7" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 45475 } } peers { permanent_uuid: "c4cedeee78a4476b802e4bad356c31a2" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 45575 } } peers { permanent_uuid: "bcda2fddf1674c178cb2988bbc77d005" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 44023 } } }
I20260806 02:52:00.605599 28988 tablet_replica.cc:442] T 2fe023dc39e643de8c07ad4d2c7e95e7 P bcda2fddf1674c178cb2988bbc77d005: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:52:00.605463 28992 tablet_replica.cc:439] T 2fe023dc39e643de8c07ad4d2c7e95e7 P c4cedeee78a4476b802e4bad356c31a2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d558e8d78f684854947a74cd5718a5c7" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 45475 } } peers { permanent_uuid: "c4cedeee78a4476b802e4bad356c31a2" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 45575 } } peers { permanent_uuid: "bcda2fddf1674c178cb2988bbc77d005" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 44023 } } }
I20260806 02:52:00.605760 28992 tablet_replica.cc:442] T 2fe023dc39e643de8c07ad4d2c7e95e7 P c4cedeee78a4476b802e4bad356c31a2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:52:00.606894 28983 ts_tablet_manager.cc:1428] T 2fe023dc39e643de8c07ad4d2c7e95e7 P c4cedeee78a4476b802e4bad356c31a2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 02:52:00.607759 28980 ts_tablet_manager.cc:1428] T 2fe023dc39e643de8c07ad4d2c7e95e7 P bcda2fddf1674c178cb2988bbc77d005: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20260806 02:52:00.608615 28979 ts_tablet_manager.cc:1428] T 2fe023dc39e643de8c07ad4d2c7e95e7 P d558e8d78f684854947a74cd5718a5c7: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260806 02:52:00.806720 28988 raft_consensus.cc:482] T 2fe023dc39e643de8c07ad4d2c7e95e7 P bcda2fddf1674c178cb2988bbc77d005 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:52:00.806985 28988 raft_consensus.cc:504] T 2fe023dc39e643de8c07ad4d2c7e95e7 P bcda2fddf1674c178cb2988bbc77d005 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d558e8d78f684854947a74cd5718a5c7" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 45475 } } peers { permanent_uuid: "c4cedeee78a4476b802e4bad356c31a2" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 45575 } } peers { permanent_uuid: "bcda2fddf1674c178cb2988bbc77d005" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 44023 } }
I20260806 02:52:00.808866 28988 leader_election.cc:288] T 2fe023dc39e643de8c07ad4d2c7e95e7 P bcda2fddf1674c178cb2988bbc77d005 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d558e8d78f684854947a74cd5718a5c7 (127.25.211.129:45475), c4cedeee78a4476b802e4bad356c31a2 (127.25.211.130:45575)
I20260806 02:52:00.828152 28972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:00.809543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:57121 (local address 127.25.211.129:45475)
0806 02:52:00.809810 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:00.809821 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:00.821154 (+ 11333us) server_negotiation.cc:407] Connection header received
0806 02:52:00.821577 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:00.821583 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:00.821666 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:00.821766 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:52:00.822359 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:00.823224 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:00.825731 (+ 2507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:00.827032 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:00.827212 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:00.827289 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:00.827646 (+ 357us) server_negotiation.cc:299] Negotiation successful
0806 02:52:00.827785 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260806 02:52:00.828639 29049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:00.809110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.130:45575 (local address 127.25.211.131:53491)
0806 02:52:00.821311 (+ 12201us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:00.821338 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:00.821485 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:00.821957 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:00.821960 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:00.821980 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:52:00.822429 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:00.822443 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:00.823872 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:00.823877 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:00.826112 (+ 2235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:00.826126 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:00.827669 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:00.827676 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:00.827887 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:00.827923 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 02:52:00.828013 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 02:52:00.828251 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12060,"mutex_wait_us":729,"thread_start_us":11153,"threads_started":1}
I20260806 02:52:00.828737 29050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:00.809412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:53491 (local address 127.25.211.130:45575)
0806 02:52:00.820989 (+ 11577us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:00.820996 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:00.821460 (+ 464us) server_negotiation.cc:407] Connection header received
0806 02:52:00.821629 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:00.821634 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:00.821704 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:00.821792 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:52:00.822607 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:00.823720 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:00.826295 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:00.827340 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:00.827575 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:00.827647 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:00.828162 (+ 515us) server_negotiation.cc:299] Negotiation successful
0806 02:52:00.828372 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11455,"thread_start_us":11367,"threads_started":1}
I20260806 02:52:00.829370 29051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:00.809110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:45475 (local address 127.25.211.131:57121)
0806 02:52:00.820988 (+ 11878us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:00.821013 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:00.821380 (+ 367us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:00.821792 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:00.821797 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:00.821822 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:52:00.822147 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:00.822192 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:00.823358 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:00.823363 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:00.825478 (+ 2115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:00.825496 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:00.827233 (+ 1737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:00.827240 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:00.827421 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:00.827459 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 02:52:00.828702 (+ 1243us) client_negotiation.cc:240] Negotiation successful
0806 02:52:00.829017 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11726,"thread_start_us":12047,"threads_started":1}
I20260806 02:52:00.829617 28673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fe023dc39e643de8c07ad4d2c7e95e7" candidate_uuid: "bcda2fddf1674c178cb2988bbc77d005" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4cedeee78a4476b802e4bad356c31a2" is_pre_election: true
I20260806 02:52:00.829909 28673 raft_consensus.cc:2454] T 2fe023dc39e643de8c07ad4d2c7e95e7 P c4cedeee78a4476b802e4bad356c31a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcda2fddf1674c178cb2988bbc77d005 in term 0.
I20260806 02:52:00.830011 28458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fe023dc39e643de8c07ad4d2c7e95e7" candidate_uuid: "bcda2fddf1674c178cb2988bbc77d005" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d558e8d78f684854947a74cd5718a5c7" is_pre_election: true
I20260806 02:52:00.830343 28458 raft_consensus.cc:2454] T 2fe023dc39e643de8c07ad4d2c7e95e7 P d558e8d78f684854947a74cd5718a5c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcda2fddf1674c178cb2988bbc77d005 in term 0.
I20260806 02:52:00.830529 28778 leader_election.cc:302] T 2fe023dc39e643de8c07ad4d2c7e95e7 P bcda2fddf1674c178cb2988bbc77d005 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bcda2fddf1674c178cb2988bbc77d005, c4cedeee78a4476b802e4bad356c31a2; no voters:
I20260806 02:52:00.831056 28988 raft_consensus.cc:2781] T 2fe023dc39e643de8c07ad4d2c7e95e7 P bcda2fddf1674c178cb2988bbc77d005 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:52:00.831199 28988 raft_consensus.cc:482] T 2fe023dc39e643de8c07ad4d2c7e95e7 P bcda2fddf1674c178cb2988bbc77d005 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:52:00.831305 28988 raft_consensus.cc:3037] T 2fe023dc39e643de8c07ad4d2c7e95e7 P bcda2fddf1674c178cb2988bbc77d005 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:00.834972 28988 raft_consensus.cc:504] T 2fe023dc39e643de8c07ad4d2c7e95e7 P bcda2fddf1674c178cb2988bbc77d005 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d558e8d78f684854947a74cd5718a5c7" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 45475 } } peers { permanent_uuid: "c4cedeee78a4476b802e4bad356c31a2" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 45575 } } peers { permanent_uuid: "bcda2fddf1674c178cb2988bbc77d005" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 44023 } }
I20260806 02:52:00.835983 28988 leader_election.cc:288] T 2fe023dc39e643de8c07ad4d2c7e95e7 P bcda2fddf1674c178cb2988bbc77d005 [CANDIDATE]: Term 1 election: Requested vote from peers d558e8d78f684854947a74cd5718a5c7 (127.25.211.129:45475), c4cedeee78a4476b802e4bad356c31a2 (127.25.211.130:45575)
I20260806 02:52:00.836787 28673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fe023dc39e643de8c07ad4d2c7e95e7" candidate_uuid: "bcda2fddf1674c178cb2988bbc77d005" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4cedeee78a4476b802e4bad356c31a2"
I20260806 02:52:00.837054 28673 raft_consensus.cc:3037] T 2fe023dc39e643de8c07ad4d2c7e95e7 P c4cedeee78a4476b802e4bad356c31a2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:00.837033 28458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fe023dc39e643de8c07ad4d2c7e95e7" candidate_uuid: "bcda2fddf1674c178cb2988bbc77d005" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d558e8d78f684854947a74cd5718a5c7"
I20260806 02:52:00.837246 28458 raft_consensus.cc:3037] T 2fe023dc39e643de8c07ad4d2c7e95e7 P d558e8d78f684854947a74cd5718a5c7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:00.839643 28458 raft_consensus.cc:2454] T 2fe023dc39e643de8c07ad4d2c7e95e7 P d558e8d78f684854947a74cd5718a5c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcda2fddf1674c178cb2988bbc77d005 in term 1.
I20260806 02:52:00.839836 28673 raft_consensus.cc:2454] T 2fe023dc39e643de8c07ad4d2c7e95e7 P c4cedeee78a4476b802e4bad356c31a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcda2fddf1674c178cb2988bbc77d005 in term 1.
I20260806 02:52:00.840139 28779 leader_election.cc:302] T 2fe023dc39e643de8c07ad4d2c7e95e7 P bcda2fddf1674c178cb2988bbc77d005 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bcda2fddf1674c178cb2988bbc77d005, d558e8d78f684854947a74cd5718a5c7; no voters:
I20260806 02:52:00.840531 28988 raft_consensus.cc:2781] T 2fe023dc39e643de8c07ad4d2c7e95e7 P bcda2fddf1674c178cb2988bbc77d005 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:52:00.841037 28988 raft_consensus.cc:686] T 2fe023dc39e643de8c07ad4d2c7e95e7 P bcda2fddf1674c178cb2988bbc77d005 [term 1 LEADER]: Becoming Leader. State: Replica: bcda2fddf1674c178cb2988bbc77d005, State: Running, Role: LEADER
I20260806 02:52:00.841503 28988 consensus_queue.cc:237] T 2fe023dc39e643de8c07ad4d2c7e95e7 P bcda2fddf1674c178cb2988bbc77d005 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d558e8d78f684854947a74cd5718a5c7" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 45475 } } peers { permanent_uuid: "c4cedeee78a4476b802e4bad356c31a2" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 45575 } } peers { permanent_uuid: "bcda2fddf1674c178cb2988bbc77d005" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 44023 } }
I20260806 02:52:00.843701 29053 tablet_replica.cc:439] T 2fe023dc39e643de8c07ad4d2c7e95e7 P bcda2fddf1674c178cb2988bbc77d005: TxnStatusTablet state changed. Reason: New leader bcda2fddf1674c178cb2988bbc77d005. Latest consensus state: current_term: 1 leader_uuid: "bcda2fddf1674c178cb2988bbc77d005" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d558e8d78f684854947a74cd5718a5c7" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 45475 } } peers { permanent_uuid: "c4cedeee78a4476b802e4bad356c31a2" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 45575 } } peers { permanent_uuid: "bcda2fddf1674c178cb2988bbc77d005" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 44023 } } }
I20260806 02:52:00.844156 29053 tablet_replica.cc:442] T 2fe023dc39e643de8c07ad4d2c7e95e7 P bcda2fddf1674c178cb2988bbc77d005: This TxnStatusTablet replica's current role is: LEADER
I20260806 02:52:00.845001 29055 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 02:52:00.845266 29055 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 02:52:00.845841 28205 catalog_manager.cc:5495] T 2fe023dc39e643de8c07ad4d2c7e95e7 P bcda2fddf1674c178cb2988bbc77d005 reported cstate change: term changed from 0 to 1, leader changed from <none> to bcda2fddf1674c178cb2988bbc77d005 (127.25.211.131). New cstate: current_term: 1 leader_uuid: "bcda2fddf1674c178cb2988bbc77d005" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d558e8d78f684854947a74cd5718a5c7" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 45475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4cedeee78a4476b802e4bad356c31a2" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 45575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcda2fddf1674c178cb2988bbc77d005" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 44023 } health_report { overall_health: HEALTHY } } }
I20260806 02:52:00.910578 29044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:00.904210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.131:44023 (local address 127.0.0.1:49298)
0806 02:52:00.904475 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:00.904490 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:00.904639 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:00.905124 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:00.905129 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:00.905145 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:52:00.905409 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:00.905419 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:00.906667 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:00.906672 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:00.907863 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:00.907878 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:00.908724 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:00.908730 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:00.908901 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:00.908970 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:52:00.909937 (+ 967us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:52:00.909965 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 02:52:00.910055 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 02:52:00.910271 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260806 02:52:00.910665 28975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:00.903894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49298 (local address 127.25.211.131:44023)
0806 02:52:00.904055 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:00.904059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:00.904578 (+ 519us) server_negotiation.cc:407] Connection header received
0806 02:52:00.904860 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:00.904864 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:00.904952 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:00.905076 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:52:00.905576 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:00.906516 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:00.908023 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:00.908552 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:00.908675 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:00.909184 (+ 509us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:52:00.909800 (+ 616us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:52:00.909959 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:00.910092 (+ 133us) server_negotiation.cc:299] Negotiation successful
0806 02:52:00.910217 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 02:52:00.916580 28673 raft_consensus.cc:1261] T 2fe023dc39e643de8c07ad4d2c7e95e7 P c4cedeee78a4476b802e4bad356c31a2 [term 1 FOLLOWER]: Refusing update from remote peer bcda2fddf1674c178cb2988bbc77d005: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:52:00.917163 28458 raft_consensus.cc:1261] T 2fe023dc39e643de8c07ad4d2c7e95e7 P d558e8d78f684854947a74cd5718a5c7 [term 1 FOLLOWER]: Refusing update from remote peer bcda2fddf1674c178cb2988bbc77d005: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:52:00.917584 29053 consensus_queue.cc:1035] T 2fe023dc39e643de8c07ad4d2c7e95e7 P bcda2fddf1674c178cb2988bbc77d005 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4cedeee78a4476b802e4bad356c31a2" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 45575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:52:00.918081 28988 consensus_queue.cc:1035] T 2fe023dc39e643de8c07ad4d2c7e95e7 P bcda2fddf1674c178cb2988bbc77d005 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d558e8d78f684854947a74cd5718a5c7" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 45475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:52:00.932494 29011 tablet_replica.cc:439] T 2fe023dc39e643de8c07ad4d2c7e95e7 P d558e8d78f684854947a74cd5718a5c7: TxnStatusTablet state changed. Reason: New leader bcda2fddf1674c178cb2988bbc77d005. Latest consensus state: current_term: 1 leader_uuid: "bcda2fddf1674c178cb2988bbc77d005" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d558e8d78f684854947a74cd5718a5c7" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 45475 } } peers { permanent_uuid: "c4cedeee78a4476b802e4bad356c31a2" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 45575 } } peers { permanent_uuid: "bcda2fddf1674c178cb2988bbc77d005" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 44023 } } }
I20260806 02:52:00.933044 29011 tablet_replica.cc:442] T 2fe023dc39e643de8c07ad4d2c7e95e7 P d558e8d78f684854947a74cd5718a5c7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:52:00.937575 29059 mvcc.cc:205] Tried to move back new op lower bound from 7315393416866816000 to 7315393416571482112. Current Snapshot: MvccSnapshot[applied={T|T < 7315393416866816000}]
I20260806 02:52:00.948426 28988 tablet_replica.cc:439] T 2fe023dc39e643de8c07ad4d2c7e95e7 P bcda2fddf1674c178cb2988bbc77d005: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bcda2fddf1674c178cb2988bbc77d005" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d558e8d78f684854947a74cd5718a5c7" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 45475 } } peers { permanent_uuid: "c4cedeee78a4476b802e4bad356c31a2" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 45575 } } peers { permanent_uuid: "bcda2fddf1674c178cb2988bbc77d005" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 44023 } } }
I20260806 02:52:00.948866 28988 tablet_replica.cc:442] T 2fe023dc39e643de8c07ad4d2c7e95e7 P bcda2fddf1674c178cb2988bbc77d005: This TxnStatusTablet replica's current role is: LEADER
I20260806 02:52:00.952684 28992 tablet_replica.cc:439] T 2fe023dc39e643de8c07ad4d2c7e95e7 P c4cedeee78a4476b802e4bad356c31a2: TxnStatusTablet state changed. Reason: New leader bcda2fddf1674c178cb2988bbc77d005. Latest consensus state: current_term: 1 leader_uuid: "bcda2fddf1674c178cb2988bbc77d005" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d558e8d78f684854947a74cd5718a5c7" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 45475 } } peers { permanent_uuid: "c4cedeee78a4476b802e4bad356c31a2" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 45575 } } peers { permanent_uuid: "bcda2fddf1674c178cb2988bbc77d005" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 44023 } } }
I20260806 02:52:00.953030 28992 tablet_replica.cc:442] T 2fe023dc39e643de8c07ad4d2c7e95e7 P c4cedeee78a4476b802e4bad356c31a2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:52:00.954946 29011 tablet_replica.cc:439] T 2fe023dc39e643de8c07ad4d2c7e95e7 P d558e8d78f684854947a74cd5718a5c7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bcda2fddf1674c178cb2988bbc77d005" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d558e8d78f684854947a74cd5718a5c7" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 45475 } } peers { permanent_uuid: "c4cedeee78a4476b802e4bad356c31a2" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 45575 } } peers { permanent_uuid: "bcda2fddf1674c178cb2988bbc77d005" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 44023 } } }
I20260806 02:52:00.955209 29011 tablet_replica.cc:442] T 2fe023dc39e643de8c07ad4d2c7e95e7 P d558e8d78f684854947a74cd5718a5c7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:52:00.958289 29053 tablet_replica.cc:439] T 2fe023dc39e643de8c07ad4d2c7e95e7 P bcda2fddf1674c178cb2988bbc77d005: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bcda2fddf1674c178cb2988bbc77d005" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d558e8d78f684854947a74cd5718a5c7" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 45475 } } peers { permanent_uuid: "c4cedeee78a4476b802e4bad356c31a2" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 45575 } } peers { permanent_uuid: "bcda2fddf1674c178cb2988bbc77d005" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 44023 } } }
I20260806 02:52:00.958673 29053 tablet_replica.cc:442] T 2fe023dc39e643de8c07ad4d2c7e95e7 P bcda2fddf1674c178cb2988bbc77d005: This TxnStatusTablet replica's current role is: LEADER
I20260806 02:52:00.963277 28992 tablet_replica.cc:439] T 2fe023dc39e643de8c07ad4d2c7e95e7 P c4cedeee78a4476b802e4bad356c31a2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bcda2fddf1674c178cb2988bbc77d005" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d558e8d78f684854947a74cd5718a5c7" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 45475 } } peers { permanent_uuid: "c4cedeee78a4476b802e4bad356c31a2" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 45575 } } peers { permanent_uuid: "bcda2fddf1674c178cb2988bbc77d005" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 44023 } } }
I20260806 02:52:00.963631 28992 tablet_replica.cc:442] T 2fe023dc39e643de8c07ad4d2c7e95e7 P c4cedeee78a4476b802e4bad356c31a2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:52:00.972072 29050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:00.963362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53890 (local address 127.25.211.130:45575)
0806 02:52:00.964453 (+ 1091us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:00.964459 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:00.964599 (+ 140us) server_negotiation.cc:407] Connection header received
0806 02:52:00.964954 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:00.964960 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:00.965089 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:00.965234 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:52:00.966010 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:00.967587 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:00.969296 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:00.969750 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:00.969869 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:00.970620 (+ 751us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:52:00.971344 (+ 724us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:52:00.971490 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:00.971613 (+ 123us) server_negotiation.cc:299] Negotiation successful
0806 02:52:00.971705 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":961}
I20260806 02:52:00.984483 28992 consensus_queue.cc:1035] T d4beecde58764f6695a2e097020d59ab P c4cedeee78a4476b802e4bad356c31a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d558e8d78f684854947a74cd5718a5c7" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 45475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:52:00.992302 28975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:00.984419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:53663 (local address 127.25.211.131:44023)
0806 02:52:00.984665 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:00.984671 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:00.985156 (+ 485us) server_negotiation.cc:407] Connection header received
0806 02:52:00.985559 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:00.985566 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:00.985655 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:00.985711 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:52:00.986666 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:00.987824 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:00.989590 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:00.990458 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:00.990564 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:00.991317 (+ 753us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:52:00.991467 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:52:00.991540 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:00.991839 (+ 299us) server_negotiation.cc:299] Negotiation successful
0806 02:52:00.991938 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 02:52:00.993871 28992 consensus_queue.cc:1035] T d4beecde58764f6695a2e097020d59ab P c4cedeee78a4476b802e4bad356c31a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcda2fddf1674c178cb2988bbc77d005" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 44023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 02:52:00.995949 29078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:00.984160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.131:44023 (local address 127.25.211.130:53663)
0806 02:52:00.985047 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:00.985068 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:00.985345 (+ 277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:00.985890 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:00.985895 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:00.985909 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:52:00.986458 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:00.986474 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:00.987990 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:00.987994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:00.989405 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:00.989420 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:00.990763 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:00.990769 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:00.990962 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:00.991052 (+ 90us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:52:00.991676 (+ 624us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:52:00.991687 (+ 11us) client_negotiation.cc:769] Sending connection context
0806 02:52:00.992854 (+ 1167us) client_negotiation.cc:240] Negotiation successful
0806 02:52:00.993013 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":1227,"threads_started":1}
I20260806 02:52:01.063454 29050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:01.052702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:60385 (local address 127.25.211.130:45575)
0806 02:52:01.052878 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:01.052883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:01.055178 (+ 2295us) server_negotiation.cc:407] Connection header received
0806 02:52:01.055555 (+ 377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:01.055561 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:01.055650 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:01.055728 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:52:01.056750 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:01.058323 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:01.060467 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:01.061115 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:01.061239 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:01.061890 (+ 651us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:52:01.062102 (+ 212us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:52:01.062296 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:01.063052 (+ 756us) server_negotiation.cc:299] Negotiation successful
0806 02:52:01.063146 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 02:52:01.063671 29086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:01.052594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.130:45575 (local address 127.25.211.131:60385)
0806 02:52:01.055061 (+ 2467us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:01.055080 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:01.055389 (+ 309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:01.055772 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:01.055777 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:01.055790 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:52:01.056533 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:01.056547 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:01.058452 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:01.058457 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:01.060302 (+ 1845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:01.060320 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:01.061292 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:01.061299 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:01.061520 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:01.061611 (+ 91us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:52:01.062910 (+ 1299us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:52:01.062921 (+ 11us) client_negotiation.cc:769] Sending connection context
0806 02:52:01.063004 (+ 83us) client_negotiation.cc:240] Negotiation successful
0806 02:52:01.063220 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2094,"thread_start_us":1841,"threads_started":1}
I20260806 02:52:01.111686 28202 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:52:01.111975 28202 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:52:01.116196 28202 catalog_manager.cc:5782] T 00000000000000000000000000000000 P db26e8a240ca4210a0c3a59bf501597c: Sending DeleteTablet for 3 replicas of tablet d4beecde58764f6695a2e097020d59ab
I20260806 02:52:01.117386 28425 tablet_service.cc:1514] Processing DeleteTablet for tablet d4beecde58764f6695a2e097020d59ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:52:01 UTC) from {username='kudu', principal='kudu/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:55794
I20260806 02:52:01.117380 28853 tablet_service.cc:1514] Processing DeleteTablet for tablet d4beecde58764f6695a2e097020d59ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:52:01 UTC) from {username='kudu', principal='kudu/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:49282
I20260806 02:52:01.119160 28645 tablet_service.cc:1514] Processing DeleteTablet for tablet d4beecde58764f6695a2e097020d59ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:52:01 UTC) from {username='kudu', principal='kudu/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:53874
I20260806 02:52:01.125663 29090 tablet_replica.cc:330] T d4beecde58764f6695a2e097020d59ab P bcda2fddf1674c178cb2988bbc77d005: stopping tablet replica
I20260806 02:52:01.126039 29090 raft_consensus.cc:2229] T d4beecde58764f6695a2e097020d59ab P bcda2fddf1674c178cb2988bbc77d005 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:52:01.126511 29090 raft_consensus.cc:2258] T d4beecde58764f6695a2e097020d59ab P bcda2fddf1674c178cb2988bbc77d005 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:52:01.127871 29090 ts_tablet_manager.cc:1905] T d4beecde58764f6695a2e097020d59ab P bcda2fddf1674c178cb2988bbc77d005: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:52:01.136828 29090 ts_tablet_manager.cc:1918] T d4beecde58764f6695a2e097020d59ab P bcda2fddf1674c178cb2988bbc77d005: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:52:01.137120 29090 log.cc:1196] T d4beecde58764f6695a2e097020d59ab P bcda2fddf1674c178cb2988bbc77d005: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/ts-2/wal/wals/d4beecde58764f6695a2e097020d59ab
I20260806 02:52:01.137744 29090 ts_tablet_manager.cc:1939] T d4beecde58764f6695a2e097020d59ab P bcda2fddf1674c178cb2988bbc77d005: Deleting consensus metadata
I20260806 02:52:01.139253 28188 catalog_manager.cc:4841] TS bcda2fddf1674c178cb2988bbc77d005 (127.25.211.131:44023): tablet d4beecde58764f6695a2e097020d59ab (table test-table [id=750ed535d2dc49faaa329bc12a54ff69]) successfully deleted
I20260806 02:52:01.143236 29091 tablet_replica.cc:330] T d4beecde58764f6695a2e097020d59ab P c4cedeee78a4476b802e4bad356c31a2: stopping tablet replica
I20260806 02:52:01.143563 29091 raft_consensus.cc:2229] T d4beecde58764f6695a2e097020d59ab P c4cedeee78a4476b802e4bad356c31a2 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:52:01.144492 29091 raft_consensus.cc:2258] T d4beecde58764f6695a2e097020d59ab P c4cedeee78a4476b802e4bad356c31a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:52:01.145017 29089 tablet_replica.cc:330] T d4beecde58764f6695a2e097020d59ab P d558e8d78f684854947a74cd5718a5c7: stopping tablet replica
I20260806 02:52:01.145437 29089 raft_consensus.cc:2229] T d4beecde58764f6695a2e097020d59ab P d558e8d78f684854947a74cd5718a5c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:52:01.146768 29091 ts_tablet_manager.cc:1905] T d4beecde58764f6695a2e097020d59ab P c4cedeee78a4476b802e4bad356c31a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:52:01.147339 29089 raft_consensus.cc:2258] T d4beecde58764f6695a2e097020d59ab P d558e8d78f684854947a74cd5718a5c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:52:01.149421 29089 ts_tablet_manager.cc:1905] T d4beecde58764f6695a2e097020d59ab P d558e8d78f684854947a74cd5718a5c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:52:01.157904 29089 ts_tablet_manager.cc:1918] T d4beecde58764f6695a2e097020d59ab P d558e8d78f684854947a74cd5718a5c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:52:01.167730 29091 ts_tablet_manager.cc:1918] T d4beecde58764f6695a2e097020d59ab P c4cedeee78a4476b802e4bad356c31a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:52:01.168066 29091 log.cc:1196] T d4beecde58764f6695a2e097020d59ab P c4cedeee78a4476b802e4bad356c31a2: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/ts-1/wal/wals/d4beecde58764f6695a2e097020d59ab
I20260806 02:52:01.168905 29091 ts_tablet_manager.cc:1939] T d4beecde58764f6695a2e097020d59ab P c4cedeee78a4476b802e4bad356c31a2: Deleting consensus metadata
I20260806 02:52:01.170537 28189 catalog_manager.cc:4841] TS c4cedeee78a4476b802e4bad356c31a2 (127.25.211.130:45575): tablet d4beecde58764f6695a2e097020d59ab (table test-table [id=750ed535d2dc49faaa329bc12a54ff69]) successfully deleted
I20260806 02:52:01.170984 29089 log.cc:1196] T d4beecde58764f6695a2e097020d59ab P d558e8d78f684854947a74cd5718a5c7: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1785984707115702-26446-0/minicluster-data/ts-0/wal/wals/d4beecde58764f6695a2e097020d59ab
I20260806 02:52:01.171579 29089 ts_tablet_manager.cc:1939] T d4beecde58764f6695a2e097020d59ab P d558e8d78f684854947a74cd5718a5c7: Deleting consensus metadata
I20260806 02:52:01.172854 28190 catalog_manager.cc:4841] TS d558e8d78f684854947a74cd5718a5c7 (127.25.211.129:45475): tablet d4beecde58764f6695a2e097020d59ab (table test-table [id=750ed535d2dc49faaa329bc12a54ff69]) successfully deleted
Aug 06 02:52:01 dist-test-slave-zj8q krb5kdc[28137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984719, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.25.211.129@KRBTEST.COM
I20260806 02:52:01.189110 28972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:01.143585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55798 (local address 127.25.211.129:45475)
0806 02:52:01.143763 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:01.143770 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:01.162882 (+ 19112us) server_negotiation.cc:407] Connection header received
0806 02:52:01.166518 (+ 3636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:01.166524 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:01.166645 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:01.166741 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:01.169560 (+ 2819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:01.170944 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:01.173317 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:01.174255 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:01.174407 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:01.180033 (+ 5626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:01.180064 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:01.180088 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:01.180149 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:01.184843 (+ 4694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:01.185968 (+ 1125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:01.185978 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:01.185982 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:01.186208 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:01.186713 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:01.186718 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:01.186721 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:01.187269 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:01.187788 (+ 519us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:01.188466 (+ 678us) server_negotiation.cc:299] Negotiation successful
0806 02:52:01.188727 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260806 02:52:01.189911 28526 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55798
I20260806 02:52:01.218748 29097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:01.194635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44780 (local address 127.25.211.190:41109)
0806 02:52:01.195204 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:01.195212 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:01.198818 (+ 3606us) server_negotiation.cc:407] Connection header received
0806 02:52:01.199051 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:01.199057 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:01.199151 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:01.199284 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:01.201688 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:01.203118 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:01.205226 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:01.206112 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:01.206268 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:01.209984 (+ 3716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:01.209992 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:01.209998 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:01.210068 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:01.214439 (+ 4371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:01.215500 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:01.215507 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:01.215511 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:01.215727 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:01.216288 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:01.216294 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:01.216298 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:01.216786 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:01.217042 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:01.217914 (+ 872us) server_negotiation.cc:299] Negotiation successful
0806 02:52:01.218200 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":251,"threads_started":1}
W20260806 02:52:01.220476 28202 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44780
I20260806 02:52:01.223951 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 28322
I20260806 02:52:01.258601 28972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:01.226247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55808 (local address 127.25.211.129:45475)
0806 02:52:01.226584 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:01.226590 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:01.237096 (+ 10506us) server_negotiation.cc:407] Connection header received
0806 02:52:01.237331 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:01.237338 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:01.237437 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:01.237563 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:01.243646 (+ 6083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:01.244803 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:01.246849 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:01.247543 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:01.247689 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:01.251393 (+ 3704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:01.251400 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:01.251406 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:01.251457 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:01.255110 (+ 3653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:01.256102 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:01.256108 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:01.256112 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:01.256294 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:01.256790 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:01.256795 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:01.256798 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:01.257167 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:01.257467 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:01.257894 (+ 427us) server_negotiation.cc:299] Negotiation successful
0806 02:52:01.258217 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191}
W20260806 02:52:01.259532 28526 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55808
W20260806 02:52:01.260314 26446 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 28322: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 02:52:01.260576 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 28322
I20260806 02:52:01.305932 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 28536
Aug 06 02:52:01 dist-test-slave-zj8q krb5kdc[28137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984719, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.25.211.130@KRBTEST.COM
I20260806 02:52:01.327533 29050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:01.306886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53898 (local address 127.25.211.130:45575)
0806 02:52:01.307131 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:01.307139 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:01.307328 (+ 189us) server_negotiation.cc:407] Connection header received
0806 02:52:01.307531 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:01.307536 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:01.307625 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:01.307741 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:01.309413 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:01.310779 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:01.312665 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:01.313380 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:01.313529 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:01.319298 (+ 5769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:01.319323 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:01.319347 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:01.319404 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:01.323889 (+ 4485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:01.324839 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:01.324851 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:01.324855 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:01.325051 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:01.325545 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:01.325551 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:01.325555 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:01.326206 (+ 651us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:01.326556 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:01.326857 (+ 301us) server_negotiation.cc:299] Negotiation successful
0806 02:52:01.327145 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
W20260806 02:52:01.328675 28739 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53898
W20260806 02:52:01.329427 26446 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 28536: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 02:52:01.329589 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 28536
I20260806 02:52:01.354647 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 28751
Aug 06 02:52:01 dist-test-slave-zj8q krb5kdc[28137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984719, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.25.211.131@KRBTEST.COM
I20260806 02:52:01.371857 28975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:01.355464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49300 (local address 127.25.211.131:44023)
0806 02:52:01.355672 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:01.355678 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:01.355736 (+ 58us) server_negotiation.cc:407] Connection header received
0806 02:52:01.355926 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:01.355931 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:01.356006 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:01.356090 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:01.357372 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:01.358125 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:01.359392 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:01.359938 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:01.360067 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:01.365384 (+ 5317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:01.365402 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:01.365420 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:01.365469 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:01.368757 (+ 3288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:01.369577 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:01.369585 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:01.369587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:01.369752 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:01.370203 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:01.370209 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:01.370212 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:01.370649 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:01.371007 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:01.371259 (+ 252us) server_negotiation.cc:299] Negotiation successful
0806 02:52:01.371494 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260806 02:52:01.372637 28942 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49300
W20260806 02:52:01.373405 26446 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 28751: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 02:52:01.373558 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 28751
I20260806 02:52:01.402429 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 28155
I20260806 02:52:01.403786 28268 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:52:01.520174 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 28155
2026-08-06T02:52:01Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5184 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:52:01 dist-test-slave-zj8q krb5kdc[29106](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:52:01 dist-test-slave-zj8q krb5kdc[29106](info): set up 2 sockets
Aug 06 02:52:01 dist-test-slave-zj8q krb5kdc[29106](info): commencing operation
krb5kdc: starting...
W20260806 02:52:02.723266 26446 mini_kdc.cc:121] Time spent starting KDC: real 1.157s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Aug 06 02:52:02 dist-test-slave-zj8q krb5kdc[29106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:52:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:52:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.211.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab.
Entry for principal kudu/127.25.211.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab.
I20260806 02:52:02.950048 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:38929
--webserver_interface=127.25.211.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:42789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.211.190:38929
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:52:03.294904 29122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:52:03.295218 29122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:52:03.295308 29122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:52:03.302656 29122 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:52:03.302788 29122 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 02:52:03.302830 29122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:52:03.302892 29122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:52:03.302942 29122 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:52:03.303014 29122 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:52:03.313215 29122 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:42789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.211.190:38929
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:38929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.211.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:52:03.314543 29122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:52:03.315881 29122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:52:03.327864 29135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:03.327896 29130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:03.327808 29131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:52:03.329048 29122 server_base.cc:1034] running on GCE node
I20260806 02:52:03.329906 29122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:52:03.331441 29122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:52:03.332664 29122 hybrid_clock.cc:648] HybridClock initialized: now 1785984723332581 us; error 70 us; skew 500 ppm
I20260806 02:52:03.333025 29122 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:52:03 dist-test-slave-zj8q krb5kdc[29106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984723, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:52:03.336295 29122 init.cc:377] Logged in from keytab as kudu/127.25.211.190@KRBTEST.COM (short username kudu)
I20260806 02:52:03.337760 29122 webserver.cc:460] Webserver started at http://127.25.211.190:33637/ using document root <none> and password file <none>
I20260806 02:52:03.338505 29122 fs_manager.cc:359] Metadata directory not provided
I20260806 02:52:03.338613 29122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:52:03.338948 29122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:52:03.341956 29122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/master-0/data/instance:
uuid: "50d796ee82c3472496eacc2910ede519"
format_stamp: "Formatted at 2026-08-06 02:52:03 on dist-test-slave-zj8q"
server_key: "6a02dbbaf1d33e43d7c48effec5c9285"
server_key_iv: "3b47b6ba7806fbe4eb6d402e85955f61"
server_key_version: "encryptionkey@0"
I20260806 02:52:03.342813 29122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/master-0/wal/instance:
uuid: "50d796ee82c3472496eacc2910ede519"
format_stamp: "Formatted at 2026-08-06 02:52:03 on dist-test-slave-zj8q"
server_key: "6a02dbbaf1d33e43d7c48effec5c9285"
server_key_iv: "3b47b6ba7806fbe4eb6d402e85955f61"
server_key_version: "encryptionkey@0"
I20260806 02:52:03.348433 29122 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 02:52:03.352340 29149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:03.353845 29122 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260806 02:52:03.354012 29122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/master-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/master-0/wal
uuid: "50d796ee82c3472496eacc2910ede519"
format_stamp: "Formatted at 2026-08-06 02:52:03 on dist-test-slave-zj8q"
server_key: "6a02dbbaf1d33e43d7c48effec5c9285"
server_key_iv: "3b47b6ba7806fbe4eb6d402e85955f61"
server_key_version: "encryptionkey@0"
I20260806 02:52:03.354277 29122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:52:03.372632 29122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:52:03.377014 29122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:52:03.377279 29122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:52:03.398383 29122 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.190:38929
I20260806 02:52:03.398402 29235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.190:38929 every 8 connection(s)
I20260806 02:52:03.399801 29122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
I20260806 02:52:03.403302 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 29122
I20260806 02:52:03.403525 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/master-0/wal/instance
I20260806 02:52:03.404183 29236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:03.404407 26446 external_mini_cluster.cc:1428] Setting key 4028f190dbf91469fdeea4d5c676b8af
I20260806 02:52:03.419691 29236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50d796ee82c3472496eacc2910ede519: Bootstrap starting.
Aug 06 02:52:03 dist-test-slave-zj8q krb5kdc[29106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:52:03.423511 29236 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50d796ee82c3472496eacc2910ede519: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:03.424762 29236 log.cc:824] T 00000000000000000000000000000000 P 50d796ee82c3472496eacc2910ede519: Log is configured to *not* fsync() on all Append() calls
I20260806 02:52:03.428411 29236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50d796ee82c3472496eacc2910ede519: No bootstrap required, opened a new log
I20260806 02:52:03.428973 29243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:03.406636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48568 (local address 127.25.211.190:38929)
0806 02:52:03.407431 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:03.407446 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:03.407493 (+ 47us) server_negotiation.cc:407] Connection header received
0806 02:52:03.409303 (+ 1810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:03.409328 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:03.409773 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:03.410138 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:03.412147 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:03.413491 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:03.415264 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:03.416056 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:03.416211 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:03.421201 (+ 4990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:03.421210 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:03.421227 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:03.421272 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:03.424413 (+ 3141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:03.425334 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:03.425344 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:03.425354 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:03.425682 (+ 328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:03.426079 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:03.426088 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:03.426091 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:03.426742 (+ 651us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:03.426961 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:03.427466 (+ 505us) server_negotiation.cc:299] Negotiation successful
0806 02:52:03.427866 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":356,"threads_started":1}
I20260806 02:52:03.436455 29236 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50d796ee82c3472496eacc2910ede519 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50d796ee82c3472496eacc2910ede519" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 38929 } }
I20260806 02:52:03.436887 29236 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50d796ee82c3472496eacc2910ede519 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:03.436993 29236 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50d796ee82c3472496eacc2910ede519 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50d796ee82c3472496eacc2910ede519, State: Initialized, Role: FOLLOWER
I20260806 02:52:03.437613 29236 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50d796ee82c3472496eacc2910ede519 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50d796ee82c3472496eacc2910ede519" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 38929 } }
I20260806 02:52:03.437853 29236 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50d796ee82c3472496eacc2910ede519 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:52:03.437984 29236 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50d796ee82c3472496eacc2910ede519 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:52:03.438140 29236 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50d796ee82c3472496eacc2910ede519 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:03.440497 29236 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50d796ee82c3472496eacc2910ede519 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50d796ee82c3472496eacc2910ede519" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 38929 } }
I20260806 02:52:03.441071 29236 leader_election.cc:302] T 00000000000000000000000000000000 P 50d796ee82c3472496eacc2910ede519 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50d796ee82c3472496eacc2910ede519; no voters:
I20260806 02:52:03.441684 29236 leader_election.cc:288] T 00000000000000000000000000000000 P 50d796ee82c3472496eacc2910ede519 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:52:03.441815 29246 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50d796ee82c3472496eacc2910ede519 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:52:03.442637 29246 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50d796ee82c3472496eacc2910ede519 [term 1 LEADER]: Becoming Leader. State: Replica: 50d796ee82c3472496eacc2910ede519, State: Running, Role: LEADER
I20260806 02:52:03.443181 29246 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50d796ee82c3472496eacc2910ede519 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50d796ee82c3472496eacc2910ede519" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 38929 } }
I20260806 02:52:03.443931 29236 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50d796ee82c3472496eacc2910ede519 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:52:03.446012 29249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50d796ee82c3472496eacc2910ede519 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50d796ee82c3472496eacc2910ede519. Latest consensus state: current_term: 1 leader_uuid: "50d796ee82c3472496eacc2910ede519" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50d796ee82c3472496eacc2910ede519" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 38929 } } }
I20260806 02:52:03.446460 29249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50d796ee82c3472496eacc2910ede519 [sys.catalog]: This master's current role is: LEADER
I20260806 02:52:03.446038 29247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50d796ee82c3472496eacc2910ede519 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50d796ee82c3472496eacc2910ede519" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50d796ee82c3472496eacc2910ede519" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 38929 } } }
I20260806 02:52:03.447992 29247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50d796ee82c3472496eacc2910ede519 [sys.catalog]: This master's current role is: LEADER
I20260806 02:52:03.451265 29253 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:52:03.456689 29253 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:52:03.467078 29253 catalog_manager.cc:1347] Generated new cluster ID: 8fe02480f266463c9cacd6a6c8d5b3a7
I20260806 02:52:03.467232 29253 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:52:03.488315 29253 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:52:03.490342 29253 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:52:03.511863 29253 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50d796ee82c3472496eacc2910ede519: Generated new TSK 0
I20260806 02:52:03.512779 29253 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.211.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab.
Entry for principal kudu/127.25.211.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab.
I20260806 02:52:03.576174 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.129:0
--local_ip_for_outbound_sockets=127.25.211.129
--webserver_interface=127.25.211.129
--webserver_port=0
--tserver_master_addrs=127.25.211.190:38929
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab
--principal=kudu/127.25.211.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:42789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:52:03.932847 29276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:52:03.933318 29276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:52:03.933565 29276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:52:03.944960 29276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:52:03.945098 29276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:52:03.945197 29276 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 02:52:03.945251 29276 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 02:52:03.945320 29276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.129
I20260806 02:52:03.955926 29276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:42789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab
--principal=kudu/127.25.211.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.211.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.25.211.190:38929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.129
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:52:03.957432 29276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:52:03.959033 29276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:52:03.973438 29282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:03.973495 29287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:03.973474 29283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:52:03.975647 29276 server_base.cc:1034] running on GCE node
I20260806 02:52:03.976368 29276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:52:03.977421 29276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:52:03.978688 29276 hybrid_clock.cc:648] HybridClock initialized: now 1785984723978616 us; error 53 us; skew 500 ppm
I20260806 02:52:03.979061 29276 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:52:03 dist-test-slave-zj8q krb5kdc[29106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984723, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:52:03.982496 29276 init.cc:377] Logged in from keytab as kudu/127.25.211.129@KRBTEST.COM (short username kudu)
I20260806 02:52:03.984256 29276 webserver.cc:460] Webserver started at http://127.25.211.129:42605/ using document root <none> and password file <none>
I20260806 02:52:03.985005 29276 fs_manager.cc:359] Metadata directory not provided
I20260806 02:52:03.985113 29276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:52:03.985396 29276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:52:03.988538 29276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/ts-0/data/instance:
uuid: "b5272675298b46c4b9bea5414bd422b4"
format_stamp: "Formatted at 2026-08-06 02:52:03 on dist-test-slave-zj8q"
server_key: "44ca02359493c3f42251d9a7df3ef3e1"
server_key_iv: "58aed15fa697467965aaac12fc88a384"
server_key_version: "encryptionkey@0"
I20260806 02:52:03.989272 29276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/ts-0/wal/instance:
uuid: "b5272675298b46c4b9bea5414bd422b4"
format_stamp: "Formatted at 2026-08-06 02:52:03 on dist-test-slave-zj8q"
server_key: "44ca02359493c3f42251d9a7df3ef3e1"
server_key_iv: "58aed15fa697467965aaac12fc88a384"
server_key_version: "encryptionkey@0"
I20260806 02:52:03.994626 29276 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260806 02:52:03.998342 29298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:03.999768 29276 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:52:03.999941 29276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/ts-0/wal
uuid: "b5272675298b46c4b9bea5414bd422b4"
format_stamp: "Formatted at 2026-08-06 02:52:03 on dist-test-slave-zj8q"
server_key: "44ca02359493c3f42251d9a7df3ef3e1"
server_key_iv: "58aed15fa697467965aaac12fc88a384"
server_key_version: "encryptionkey@0"
I20260806 02:52:04.000154 29276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:52:04.039345 29276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:52:04.043941 29276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:52:04.044248 29276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:52:04.047331 29276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:52:04.047439 29276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:04.047554 29276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:52:04.047647 29276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 02:52:04 dist-test-slave-zj8q krb5kdc[29106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984723, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.129@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:52:04.074345 29310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:04.047818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:38733 (local address 127.25.211.190:38929)
0806 02:52:04.050801 (+ 2983us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:04.050806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:04.050827 (+ 21us) server_negotiation.cc:407] Connection header received
0806 02:52:04.053239 (+ 2412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:04.053244 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:04.053326 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:04.053430 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:04.056321 (+ 2891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:04.057262 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:04.060006 (+ 2744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:04.060595 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:04.060741 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:04.066401 (+ 5660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:04.066407 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:04.066411 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:04.066452 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:04.069616 (+ 3164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:04.071049 (+ 1433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:04.071055 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:04.071059 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:04.071258 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:04.072033 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:04.072039 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:04.072042 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:04.072447 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:04.072610 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:04.073558 (+ 948us) server_negotiation.cc:299] Negotiation successful
0806 02:52:04.073861 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2876,"thread_start_us":2767,"threads_started":1}
I20260806 02:52:04.075351 29311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:04.048726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:38929 (local address 127.25.211.129:38733)
0806 02:52:04.049791 (+ 1065us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:04.049850 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:04.052803 (+ 2953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:04.053780 (+ 977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:04.053794 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:04.054864 (+ 1070us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:04.056014 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:04.056052 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:04.057539 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:04.057545 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:04.059829 (+ 2284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:04.059852 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:04.061088 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:04.061094 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:04.061416 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:04.062267 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:52:04.062315 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:04.065986 (+ 3671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:04.069859 (+ 3873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:04.069874 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:04.069900 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:04.070827 (+ 927us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:04.071399 (+ 572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:04.071403 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:04.071405 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:04.071886 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:04.072684 (+ 798us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:04.072691 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:04.073109 (+ 418us) client_negotiation.cc:769] Sending connection context
0806 02:52:04.073365 (+ 256us) client_negotiation.cc:240] Negotiation successful
0806 02:52:04.073862 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"thread_start_us":436,"threads_started":1}
I20260806 02:52:04.101665 29417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:04.085811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:38929 (local address 127.25.211.129:49151)
0806 02:52:04.086470 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:04.086490 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:04.086650 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:04.087059 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:04.087064 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:04.087732 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:04.088175 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:04.088190 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:04.089268 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:04.089274 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:04.091076 (+ 1802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:04.091099 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:04.092235 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:04.092244 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:04.092436 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:04.093149 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:52:04.093173 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:04.094384 (+ 1211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:04.097997 (+ 3613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:04.098004 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:04.098008 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:04.098754 (+ 746us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:04.099313 (+ 559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:04.099316 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:04.099319 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:04.099479 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:04.100263 (+ 784us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:04.100269 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:04.100499 (+ 230us) client_negotiation.cc:769] Sending connection context
0806 02:52:04.100696 (+ 197us) client_negotiation.cc:240] Negotiation successful
0806 02:52:04.101079 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":375,"threads_started":1}
I20260806 02:52:04.101958 29310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:04.085665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:49151 (local address 127.25.211.190:38929)
0806 02:52:04.085850 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:04.085855 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:04.086620 (+ 765us) server_negotiation.cc:407] Connection header received
0806 02:52:04.086785 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:04.086790 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:04.086881 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:04.086985 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:04.088366 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:04.089137 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:04.091272 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:04.091778 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:04.091915 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:04.094623 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:04.094628 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:04.094631 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:04.094669 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:04.097802 (+ 3133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:04.098934 (+ 1132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:04.098940 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:04.098944 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:04.099155 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:04.099643 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:04.099649 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:04.099652 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:04.100013 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:04.100185 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:04.101356 (+ 1171us) server_negotiation.cc:299] Negotiation successful
0806 02:52:04.101626 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 02:52:04.107101 29276 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.129:33379
I20260806 02:52:04.107136 29474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.129:33379 every 8 connection(s)
I20260806 02:52:04.108425 29276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
I20260806 02:52:04.110730 29476 heartbeater.cc:347] Connected to a master server at 127.25.211.190:38929
I20260806 02:52:04.110981 29476 heartbeater.cc:464] Registering TS with master...
I20260806 02:52:04.111690 29476 heartbeater.cc:511] Master 127.25.211.190:38929 requested a full tablet report, sending...
I20260806 02:52:04.113056 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 29276
I20260806 02:52:04.113265 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/ts-0/wal/instance
I20260806 02:52:04.113914 26446 external_mini_cluster.cc:1428] Setting key 6ee0281fbeb9e9de087bf38df514d9cb
I20260806 02:52:04.113885 29172 ts_manager.cc:194] Registered new tserver with Master: b5272675298b46c4b9bea5414bd422b4 (127.25.211.129:33379)
I20260806 02:52:04.115948 29172 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.211.129@KRBTEST.COM'} at 127.25.211.129:38733
WARNING: no policy specified for kudu/127.25.211.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab.
Entry for principal kudu/127.25.211.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab.
I20260806 02:52:04.164011 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.130:0
--local_ip_for_outbound_sockets=127.25.211.130
--webserver_interface=127.25.211.130
--webserver_port=0
--tserver_master_addrs=127.25.211.190:38929
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab
--principal=kudu/127.25.211.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:42789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:52:04.500715 29483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:52:04.501025 29483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:52:04.501158 29483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:52:04.508741 29483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:52:04.508872 29483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:52:04.508987 29483 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 02:52:04.509063 29483 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 02:52:04.509150 29483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.130
I20260806 02:52:04.519908 29483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:42789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab
--principal=kudu/127.25.211.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.211.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.25.211.190:38929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.130
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:52:04.521328 29483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:52:04.522768 29483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:52:04.536480 29491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:04.536597 29488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:04.536523 29489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:52:04.537145 29483 server_base.cc:1034] running on GCE node
I20260806 02:52:04.537869 29483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:52:04.538977 29483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:52:04.540199 29483 hybrid_clock.cc:648] HybridClock initialized: now 1785984724540123 us; error 75 us; skew 500 ppm
I20260806 02:52:04.540531 29483 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:52:04 dist-test-slave-zj8q krb5kdc[29106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984724, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:52:04.546967 29483 init.cc:377] Logged in from keytab as kudu/127.25.211.130@KRBTEST.COM (short username kudu)
I20260806 02:52:04.548596 29483 webserver.cc:460] Webserver started at http://127.25.211.130:34661/ using document root <none> and password file <none>
I20260806 02:52:04.549371 29483 fs_manager.cc:359] Metadata directory not provided
I20260806 02:52:04.549504 29483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:52:04.549845 29483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:52:04.552960 29483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/ts-1/data/instance:
uuid: "9d1c9b0b0124405398afc3033537cf76"
format_stamp: "Formatted at 2026-08-06 02:52:04 on dist-test-slave-zj8q"
server_key: "4f31e01e236819a125401acaf3e8c356"
server_key_iv: "aa4807bd93d95062cefbce9cb5251839"
server_key_version: "encryptionkey@0"
I20260806 02:52:04.553735 29483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/ts-1/wal/instance:
uuid: "9d1c9b0b0124405398afc3033537cf76"
format_stamp: "Formatted at 2026-08-06 02:52:04 on dist-test-slave-zj8q"
server_key: "4f31e01e236819a125401acaf3e8c356"
server_key_iv: "aa4807bd93d95062cefbce9cb5251839"
server_key_version: "encryptionkey@0"
I20260806 02:52:04.559496 29483 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260806 02:52:04.563475 29501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:04.565107 29483 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 02:52:04.565310 29483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/ts-1/wal
uuid: "9d1c9b0b0124405398afc3033537cf76"
format_stamp: "Formatted at 2026-08-06 02:52:04 on dist-test-slave-zj8q"
server_key: "4f31e01e236819a125401acaf3e8c356"
server_key_iv: "aa4807bd93d95062cefbce9cb5251839"
server_key_version: "encryptionkey@0"
I20260806 02:52:04.565555 29483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:52:04.578267 29483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:52:04.582944 29483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:52:04.583236 29483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:52:04.586467 29483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:52:04.586601 29483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:04.586753 29483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:52:04.586846 29483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 02:52:04 dist-test-slave-zj8q krb5kdc[29106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984724, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.130@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:52:04.610616 29310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:04.586908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:46337 (local address 127.25.211.190:38929)
0806 02:52:04.587091 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:04.587098 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:04.588637 (+ 1539us) server_negotiation.cc:407] Connection header received
0806 02:52:04.590787 (+ 2150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:04.590794 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:04.590876 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:04.590965 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:04.593167 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:04.594140 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:04.596884 (+ 2744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:04.597668 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:04.597846 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:04.603256 (+ 5410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:04.603264 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:04.603269 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:04.603321 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:04.606991 (+ 3670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:04.607893 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:04.607899 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:04.607903 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:04.608084 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:04.608561 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:04.608566 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:04.608569 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:04.608989 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:04.609170 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:04.609968 (+ 798us) server_negotiation.cc:299] Negotiation successful
0806 02:52:04.610247 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 02:52:04.611780 29514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:04.587643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:38929 (local address 127.25.211.130:46337)
0806 02:52:04.588467 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:04.588512 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:04.590406 (+ 1894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:04.591244 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:04.591256 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:04.592045 (+ 789us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:04.592966 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:04.592988 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:04.594334 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:04.594341 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:04.596628 (+ 2287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:04.596649 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:04.597935 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:04.597941 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:04.598281 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:04.599075 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:52:04.599119 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:04.602891 (+ 3772us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:04.607215 (+ 4324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:04.607226 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:04.607246 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:04.607764 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:04.608226 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:04.608230 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:04.608232 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:04.608455 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:04.609157 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:04.609164 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:04.609558 (+ 394us) client_negotiation.cc:769] Sending connection context
0806 02:52:04.609863 (+ 305us) client_negotiation.cc:240] Negotiation successful
0806 02:52:04.610363 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":367,"threads_started":1}
I20260806 02:52:04.635458 29609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:04.619002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:38929 (local address 127.25.211.130:40511)
0806 02:52:04.619661 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:04.619696 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:04.619890 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:04.620292 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:04.620296 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:04.620946 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:04.621396 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:04.621412 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:04.622886 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:04.622892 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:04.624457 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:04.624475 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:04.625782 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:04.625790 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:04.625999 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:04.626735 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:52:04.626758 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:04.628035 (+ 1277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:04.632028 (+ 3993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:04.632034 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:04.632039 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:04.632650 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:04.633216 (+ 566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:04.633221 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:04.633224 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:04.633405 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:04.634130 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:04.634139 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:04.634405 (+ 266us) client_negotiation.cc:769] Sending connection context
0806 02:52:04.634590 (+ 185us) client_negotiation.cc:240] Negotiation successful
0806 02:52:04.634977 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":320,"threads_started":1}
I20260806 02:52:04.636716 29310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:04.618757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:40511 (local address 127.25.211.190:38929)
0806 02:52:04.618946 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:04.618953 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:04.619819 (+ 866us) server_negotiation.cc:407] Connection header received
0806 02:52:04.620041 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:04.620047 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:04.620151 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:04.620257 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:04.621615 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:04.622707 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:04.624623 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:04.625332 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:04.625470 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:04.628322 (+ 2852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:04.628329 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:04.628333 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:04.628375 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:04.631855 (+ 3480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:04.632832 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:04.632837 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:04.632840 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:04.633054 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:04.633540 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:04.633546 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:04.633550 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:04.633927 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:04.634142 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:04.636108 (+ 1966us) server_negotiation.cc:299] Negotiation successful
0806 02:52:04.636390 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 02:52:04.636930 29483 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.130:46561
I20260806 02:52:04.636965 29679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.130:46561 every 8 connection(s)
I20260806 02:52:04.638322 29483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
I20260806 02:52:04.640931 29680 heartbeater.cc:347] Connected to a master server at 127.25.211.190:38929
I20260806 02:52:04.641177 29680 heartbeater.cc:464] Registering TS with master...
I20260806 02:52:04.641773 29680 heartbeater.cc:511] Master 127.25.211.190:38929 requested a full tablet report, sending...
I20260806 02:52:04.643085 29172 ts_manager.cc:194] Registered new tserver with Master: 9d1c9b0b0124405398afc3033537cf76 (127.25.211.130:46561)
I20260806 02:52:04.643865 29172 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.211.130@KRBTEST.COM'} at 127.25.211.130:46337
I20260806 02:52:04.647864 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 29483
I20260806 02:52:04.648039 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/ts-1/wal/instance
I20260806 02:52:04.648483 26446 external_mini_cluster.cc:1428] Setting key 651bca340942338b0f6a30e0d9c2e97c
WARNING: no policy specified for kudu/127.25.211.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab.
Entry for principal kudu/127.25.211.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab.
I20260806 02:52:04.695830 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.131:0
--local_ip_for_outbound_sockets=127.25.211.131
--webserver_interface=127.25.211.131
--webserver_port=0
--tserver_master_addrs=127.25.211.190:38929
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab
--principal=kudu/127.25.211.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:42789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:52:05.091136 29684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:52:05.091471 29684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:52:05.091636 29684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:52:05.099300 29684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:52:05.099439 29684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:52:05.099570 29684 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 02:52:05.099638 29684 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 02:52:05.099749 29684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.131
I20260806 02:52:05.110482 29684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:42789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab
--principal=kudu/127.25.211.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.211.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.25.211.190:38929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.131
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:52:05.111896 29684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:52:05.113284 29684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 02:52:05.119026 29476 heartbeater.cc:503] Master 127.25.211.190:38929 was elected leader, sending a full tablet report...
W20260806 02:52:05.127103 29690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:05.127055 29691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:05.127027 29695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:52:05.127861 29684 server_base.cc:1034] running on GCE node
I20260806 02:52:05.128515 29684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:52:05.129565 29684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:52:05.130786 29684 hybrid_clock.cc:648] HybridClock initialized: now 1785984725130728 us; error 50 us; skew 500 ppm
I20260806 02:52:05.131114 29684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:52:05 dist-test-slave-zj8q krb5kdc[29106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984725, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:52:05.135051 29684 init.cc:377] Logged in from keytab as kudu/127.25.211.131@KRBTEST.COM (short username kudu)
I20260806 02:52:05.136734 29684 webserver.cc:460] Webserver started at http://127.25.211.131:36695/ using document root <none> and password file <none>
I20260806 02:52:05.137476 29684 fs_manager.cc:359] Metadata directory not provided
I20260806 02:52:05.137605 29684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:52:05.137938 29684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:52:05.141256 29684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/ts-2/data/instance:
uuid: "d3114496d23749e48a2889e0a3ceef40"
format_stamp: "Formatted at 2026-08-06 02:52:05 on dist-test-slave-zj8q"
server_key: "44107b3f779fa684a757e504adb87905"
server_key_iv: "a4deebe728299e76ab07a1e514b75b94"
server_key_version: "encryptionkey@0"
I20260806 02:52:05.142074 29684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/ts-2/wal/instance:
uuid: "d3114496d23749e48a2889e0a3ceef40"
format_stamp: "Formatted at 2026-08-06 02:52:05 on dist-test-slave-zj8q"
server_key: "44107b3f779fa684a757e504adb87905"
server_key_iv: "a4deebe728299e76ab07a1e514b75b94"
server_key_version: "encryptionkey@0"
I20260806 02:52:05.147653 29684 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260806 02:52:05.151567 29706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:05.153085 29684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260806 02:52:05.153251 29684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/ts-2/wal
uuid: "d3114496d23749e48a2889e0a3ceef40"
format_stamp: "Formatted at 2026-08-06 02:52:05 on dist-test-slave-zj8q"
server_key: "44107b3f779fa684a757e504adb87905"
server_key_iv: "a4deebe728299e76ab07a1e514b75b94"
server_key_version: "encryptionkey@0"
I20260806 02:52:05.153463 29684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:52:05.168543 29684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:52:05.172888 29684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:52:05.173173 29684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:52:05.176100 29684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:52:05.176229 29684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:05.176364 29684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:52:05.176465 29684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 02:52:05 dist-test-slave-zj8q krb5kdc[29106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984725, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.131@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:52:05.201886 29719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:05.176635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:44973 (local address 127.25.211.190:38929)
0806 02:52:05.179279 (+ 2644us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:05.179286 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:05.179313 (+ 27us) server_negotiation.cc:407] Connection header received
0806 02:52:05.181226 (+ 1913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:05.181232 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:05.181334 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:05.181447 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:05.184536 (+ 3089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:05.185555 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:05.188608 (+ 3053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:05.189141 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:05.189283 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:05.194469 (+ 5186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:05.194478 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:05.194483 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:05.194523 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:05.197781 (+ 3258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:05.198769 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:05.198775 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:05.198779 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:05.198940 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:05.199567 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:05.199572 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:05.199575 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:05.199888 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:05.200042 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:05.201378 (+ 1336us) server_negotiation.cc:299] Negotiation successful
0806 02:52:05.201583 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2506,"thread_start_us":2392,"threads_started":1}
I20260806 02:52:05.203980 29720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:05.177414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:38929 (local address 127.25.211.131:44973)
0806 02:52:05.178589 (+ 1175us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:05.178661 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:05.180876 (+ 2215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:05.181983 (+ 1107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:05.181996 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:05.183037 (+ 1041us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:05.184249 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:05.184289 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:05.185745 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:05.185756 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:05.188399 (+ 2643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:05.188447 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:05.189424 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:05.189433 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:05.189680 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:05.190491 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:52:05.190524 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:05.194021 (+ 3497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:05.197932 (+ 3911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:05.197943 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:05.197963 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:05.198633 (+ 670us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:05.199072 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:05.199082 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:05.199086 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:05.199410 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:05.200029 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:05.200038 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:05.200657 (+ 619us) client_negotiation.cc:769] Sending connection context
0806 02:52:05.201263 (+ 606us) client_negotiation.cc:240] Negotiation successful
0806 02:52:05.201979 (+ 716us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"thread_start_us":406,"threads_started":1}
I20260806 02:52:05.221230 29684 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.131:45757
I20260806 02:52:05.221267 29883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.131:45757 every 8 connection(s)
I20260806 02:52:05.222872 29684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
I20260806 02:52:05.225248 29884 heartbeater.cc:347] Connected to a master server at 127.25.211.190:38929
I20260806 02:52:05.225607 29884 heartbeater.cc:464] Registering TS with master...
I20260806 02:52:05.226054 29853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:05.210558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:38929 (local address 127.25.211.131:54023)
0806 02:52:05.211232 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:05.211253 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:05.211428 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:05.211774 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:05.211780 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:05.212381 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:05.212770 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:05.212784 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:05.213730 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:05.213735 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:05.215621 (+ 1886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:05.215646 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:05.216740 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:05.216750 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:05.216938 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:05.217626 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:52:05.217654 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:05.219062 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:05.222212 (+ 3150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:05.222218 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:05.222221 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:05.222728 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:05.223987 (+ 1259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:05.223993 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:05.223997 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:05.224184 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:05.224827 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:05.224833 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:05.225060 (+ 227us) client_negotiation.cc:769] Sending connection context
0806 02:52:05.225200 (+ 140us) client_negotiation.cc:240] Negotiation successful
0806 02:52:05.225586 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":334,"threads_started":1}
I20260806 02:52:05.226361 29719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:05.210460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:54023 (local address 127.25.211.190:38929)
0806 02:52:05.210643 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:05.210649 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:05.211366 (+ 717us) server_negotiation.cc:407] Connection header received
0806 02:52:05.211548 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:05.211552 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:05.211620 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:05.211733 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:05.212901 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:05.213605 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:05.215805 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:05.216379 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:05.216508 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:05.219265 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:05.219270 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:05.219273 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:05.219308 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:05.222053 (+ 2745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:05.222900 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:05.222906 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:05.222909 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:05.223734 (+ 825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:05.224313 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:05.224319 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:05.224323 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:05.224693 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:05.224862 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:05.225387 (+ 525us) server_negotiation.cc:299] Negotiation successful
0806 02:52:05.225759 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 02:52:05.227267 29884 heartbeater.cc:511] Master 127.25.211.190:38929 requested a full tablet report, sending...
I20260806 02:52:05.228854 29172 ts_manager.cc:194] Registered new tserver with Master: d3114496d23749e48a2889e0a3ceef40 (127.25.211.131:45757)
I20260806 02:52:05.229960 29172 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.211.131@KRBTEST.COM'} at 127.25.211.131:44973
I20260806 02:52:05.231524 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 29684
I20260806 02:52:05.231714 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/ts-2/wal/instance
I20260806 02:52:05.232295 26446 external_mini_cluster.cc:1428] Setting key 6e3a51155db58cae8d7dcf2e8792532f
I20260806 02:52:05.238291 26446 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 02:52:05 dist-test-slave-zj8q krb5kdc[29106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 02:52:05 dist-test-slave-zj8q krb5kdc[29106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984725, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:52:05.293643 29719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:05.279466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48576 (local address 127.25.211.190:38929)
0806 02:52:05.279642 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:05.279647 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:05.279948 (+ 301us) server_negotiation.cc:407] Connection header received
0806 02:52:05.280105 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:05.280109 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:05.280179 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:05.280275 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:05.281721 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:05.282750 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:05.284027 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:05.284456 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:05.284550 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:05.288427 (+ 3877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:05.288433 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:05.288436 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:05.288477 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:05.291168 (+ 2691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:05.291864 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:05.291869 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:05.291872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:05.292037 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:05.292402 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:05.292407 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:05.292409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:05.292764 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:05.293007 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:05.293194 (+ 187us) server_negotiation.cc:299] Negotiation successful
0806 02:52:05.293391 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 02:52:05.298285 29172 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48576:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:52:05.305986 29172 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:52:05.330551 29906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:05.319022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:33379 (local address 127.0.0.1:52294)
0806 02:52:05.320520 (+ 1498us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:05.320568 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:05.320738 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:05.322054 (+ 1316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:05.322060 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:05.322086 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:52:05.322622 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:05.322637 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:05.324734 (+ 2097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:05.324740 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:05.327420 (+ 2680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:05.327434 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:05.329487 (+ 2053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:05.329494 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:05.329727 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:05.329793 (+ 66us) client_negotiation.cc:769] Sending connection context
0806 02:52:05.329926 (+ 133us) client_negotiation.cc:240] Negotiation successful
0806 02:52:05.330169 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1230,"thread_start_us":3299,"threads_started":1}
I20260806 02:52:05.332027 29905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:05.319182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52294 (local address 127.25.211.129:33379)
0806 02:52:05.321348 (+ 2166us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:05.321357 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:05.321389 (+ 32us) server_negotiation.cc:407] Connection header received
0806 02:52:05.321490 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:05.321498 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:05.321767 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:05.322849 (+ 1082us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:52:05.323106 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:05.324560 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:05.327648 (+ 3088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:05.329237 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:05.331267 (+ 2030us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:05.331475 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:05.331554 (+ 79us) server_negotiation.cc:299] Negotiation successful
0806 02:52:05.331735 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2000,"thread_start_us":1819,"threads_started":1}
I20260806 02:52:05.333680 29903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:05.319523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.131:45757 (local address 127.0.0.1:60664)
0806 02:52:05.322665 (+ 3142us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:05.322691 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:05.322817 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:05.323525 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:05.323530 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:05.323557 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:52:05.323910 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:05.323921 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:05.327130 (+ 3209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:05.327135 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:05.329255 (+ 2120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:05.329271 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:05.332896 (+ 3625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:05.332903 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:05.333061 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:05.333087 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 02:52:05.333174 (+ 87us) client_negotiation.cc:240] Negotiation successful
0806 02:52:05.333311 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2984,"thread_start_us":703,"threads_started":1}
I20260806 02:52:05.333832 29908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:05.320069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60664 (local address 127.25.211.131:45757)
0806 02:52:05.321871 (+ 1802us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:05.321882 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:05.322949 (+ 1067us) server_negotiation.cc:407] Connection header received
0806 02:52:05.323054 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:05.323061 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:05.323308 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:05.323516 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:52:05.324143 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:05.325760 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:05.329450 (+ 3690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:05.332318 (+ 2868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:05.332533 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:05.332723 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:05.333251 (+ 528us) server_negotiation.cc:299] Negotiation successful
0806 02:52:05.333428 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1592,"thread_start_us":1765,"threads_started":1}
I20260806 02:52:05.336879 29380 tablet_service.cc:1467] Processing CreateTablet for tablet 4b5f5594d84b471da53480aacf727742 (DEFAULT_TABLE table=test-table [id=756b4455f70943cdbbece77adce94357]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:52:05.337059 29782 tablet_service.cc:1467] Processing CreateTablet for tablet 4b5f5594d84b471da53480aacf727742 (DEFAULT_TABLE table=test-table [id=756b4455f70943cdbbece77adce94357]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:52:05.338430 29782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b5f5594d84b471da53480aacf727742. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:05.338446 29380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b5f5594d84b471da53480aacf727742. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:05.340341 29906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:05.323975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.130:46561 (local address 127.0.0.1:41152)
0806 02:52:05.330934 (+ 6959us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:05.330952 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:05.331050 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:05.333456 (+ 2406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:05.333461 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:05.333490 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:52:05.333706 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:05.333719 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:05.335913 (+ 2194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:05.335918 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:05.337690 (+ 1772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:05.337719 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:05.339526 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:05.339532 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:05.339736 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:05.339772 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 02:52:05.339879 (+ 107us) client_negotiation.cc:240] Negotiation successful
0806 02:52:05.340028 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6864,"thread_start_us":8505,"threads_started":1}
I20260806 02:52:05.340404 29911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:05.323057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41152 (local address 127.25.211.130:46561)
0806 02:52:05.332755 (+ 9698us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:05.332763 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:05.332790 (+ 27us) server_negotiation.cc:407] Connection header received
0806 02:52:05.332894 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:05.332903 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:05.333222 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:05.334057 (+ 835us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:52:05.334295 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:05.335722 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:05.337932 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:05.339321 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:05.339519 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:05.339714 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:05.339945 (+ 231us) server_negotiation.cc:299] Negotiation successful
0806 02:52:05.340125 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8997,"thread_start_us":9120,"threads_started":1}
I20260806 02:52:05.344622 29577 tablet_service.cc:1467] Processing CreateTablet for tablet 4b5f5594d84b471da53480aacf727742 (DEFAULT_TABLE table=test-table [id=756b4455f70943cdbbece77adce94357]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:52:05.346378 29577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b5f5594d84b471da53480aacf727742. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:05.351326 29914 tablet_bootstrap.cc:492] T 4b5f5594d84b471da53480aacf727742 P d3114496d23749e48a2889e0a3ceef40: Bootstrap starting.
I20260806 02:52:05.354975 29916 tablet_bootstrap.cc:492] T 4b5f5594d84b471da53480aacf727742 P b5272675298b46c4b9bea5414bd422b4: Bootstrap starting.
I20260806 02:52:05.355463 29914 tablet_bootstrap.cc:654] T 4b5f5594d84b471da53480aacf727742 P d3114496d23749e48a2889e0a3ceef40: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:05.356954 29914 log.cc:824] T 4b5f5594d84b471da53480aacf727742 P d3114496d23749e48a2889e0a3ceef40: Log is configured to *not* fsync() on all Append() calls
I20260806 02:52:05.358978 29916 tablet_bootstrap.cc:654] T 4b5f5594d84b471da53480aacf727742 P b5272675298b46c4b9bea5414bd422b4: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:05.360203 29916 log.cc:824] T 4b5f5594d84b471da53480aacf727742 P b5272675298b46c4b9bea5414bd422b4: Log is configured to *not* fsync() on all Append() calls
I20260806 02:52:05.360816 29914 tablet_bootstrap.cc:492] T 4b5f5594d84b471da53480aacf727742 P d3114496d23749e48a2889e0a3ceef40: No bootstrap required, opened a new log
I20260806 02:52:05.361137 29914 ts_tablet_manager.cc:1397] T 4b5f5594d84b471da53480aacf727742 P d3114496d23749e48a2889e0a3ceef40: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260806 02:52:05.363125 29921 tablet_bootstrap.cc:492] T 4b5f5594d84b471da53480aacf727742 P 9d1c9b0b0124405398afc3033537cf76: Bootstrap starting.
I20260806 02:52:05.365069 29916 tablet_bootstrap.cc:492] T 4b5f5594d84b471da53480aacf727742 P b5272675298b46c4b9bea5414bd422b4: No bootstrap required, opened a new log
I20260806 02:52:05.365378 29916 ts_tablet_manager.cc:1397] T 4b5f5594d84b471da53480aacf727742 P b5272675298b46c4b9bea5414bd422b4: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260806 02:52:05.366957 29921 tablet_bootstrap.cc:654] T 4b5f5594d84b471da53480aacf727742 P 9d1c9b0b0124405398afc3033537cf76: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:05.368362 29921 log.cc:824] T 4b5f5594d84b471da53480aacf727742 P 9d1c9b0b0124405398afc3033537cf76: Log is configured to *not* fsync() on all Append() calls
I20260806 02:52:05.369815 29914 raft_consensus.cc:348] T 4b5f5594d84b471da53480aacf727742 P d3114496d23749e48a2889e0a3ceef40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5272675298b46c4b9bea5414bd422b4" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 33379 } } peers { permanent_uuid: "9d1c9b0b0124405398afc3033537cf76" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46561 } } peers { permanent_uuid: "d3114496d23749e48a2889e0a3ceef40" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45757 } }
I20260806 02:52:05.370369 29914 raft_consensus.cc:374] T 4b5f5594d84b471da53480aacf727742 P d3114496d23749e48a2889e0a3ceef40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:05.370473 29914 raft_consensus.cc:729] T 4b5f5594d84b471da53480aacf727742 P d3114496d23749e48a2889e0a3ceef40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3114496d23749e48a2889e0a3ceef40, State: Initialized, Role: FOLLOWER
I20260806 02:52:05.371279 29914 consensus_queue.cc:260] T 4b5f5594d84b471da53480aacf727742 P d3114496d23749e48a2889e0a3ceef40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5272675298b46c4b9bea5414bd422b4" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 33379 } } peers { permanent_uuid: "9d1c9b0b0124405398afc3033537cf76" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46561 } } peers { permanent_uuid: "d3114496d23749e48a2889e0a3ceef40" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45757 } }
I20260806 02:52:05.372531 29884 heartbeater.cc:503] Master 127.25.211.190:38929 was elected leader, sending a full tablet report...
I20260806 02:52:05.372747 29921 tablet_bootstrap.cc:492] T 4b5f5594d84b471da53480aacf727742 P 9d1c9b0b0124405398afc3033537cf76: No bootstrap required, opened a new log
I20260806 02:52:05.373064 29921 ts_tablet_manager.cc:1397] T 4b5f5594d84b471da53480aacf727742 P 9d1c9b0b0124405398afc3033537cf76: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260806 02:52:05.373255 29914 ts_tablet_manager.cc:1428] T 4b5f5594d84b471da53480aacf727742 P d3114496d23749e48a2889e0a3ceef40: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260806 02:52:05.375831 29916 raft_consensus.cc:348] T 4b5f5594d84b471da53480aacf727742 P b5272675298b46c4b9bea5414bd422b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5272675298b46c4b9bea5414bd422b4" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 33379 } } peers { permanent_uuid: "9d1c9b0b0124405398afc3033537cf76" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46561 } } peers { permanent_uuid: "d3114496d23749e48a2889e0a3ceef40" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45757 } }
I20260806 02:52:05.376277 29916 raft_consensus.cc:374] T 4b5f5594d84b471da53480aacf727742 P b5272675298b46c4b9bea5414bd422b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:05.376377 29916 raft_consensus.cc:729] T 4b5f5594d84b471da53480aacf727742 P b5272675298b46c4b9bea5414bd422b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5272675298b46c4b9bea5414bd422b4, State: Initialized, Role: FOLLOWER
I20260806 02:52:05.377023 29916 consensus_queue.cc:260] T 4b5f5594d84b471da53480aacf727742 P b5272675298b46c4b9bea5414bd422b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5272675298b46c4b9bea5414bd422b4" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 33379 } } peers { permanent_uuid: "9d1c9b0b0124405398afc3033537cf76" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46561 } } peers { permanent_uuid: "d3114496d23749e48a2889e0a3ceef40" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45757 } }
I20260806 02:52:05.378942 29916 ts_tablet_manager.cc:1428] T 4b5f5594d84b471da53480aacf727742 P b5272675298b46c4b9bea5414bd422b4: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260806 02:52:05.382917 29921 raft_consensus.cc:348] T 4b5f5594d84b471da53480aacf727742 P 9d1c9b0b0124405398afc3033537cf76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5272675298b46c4b9bea5414bd422b4" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 33379 } } peers { permanent_uuid: "9d1c9b0b0124405398afc3033537cf76" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46561 } } peers { permanent_uuid: "d3114496d23749e48a2889e0a3ceef40" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45757 } }
I20260806 02:52:05.383374 29921 raft_consensus.cc:374] T 4b5f5594d84b471da53480aacf727742 P 9d1c9b0b0124405398afc3033537cf76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:05.383473 29921 raft_consensus.cc:729] T 4b5f5594d84b471da53480aacf727742 P 9d1c9b0b0124405398afc3033537cf76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d1c9b0b0124405398afc3033537cf76, State: Initialized, Role: FOLLOWER
I20260806 02:52:05.384117 29921 consensus_queue.cc:260] T 4b5f5594d84b471da53480aacf727742 P 9d1c9b0b0124405398afc3033537cf76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5272675298b46c4b9bea5414bd422b4" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 33379 } } peers { permanent_uuid: "9d1c9b0b0124405398afc3033537cf76" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46561 } } peers { permanent_uuid: "d3114496d23749e48a2889e0a3ceef40" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45757 } }
I20260806 02:52:05.385119 29680 heartbeater.cc:503] Master 127.25.211.190:38929 was elected leader, sending a full tablet report...
I20260806 02:52:05.385839 29921 ts_tablet_manager.cc:1428] T 4b5f5594d84b471da53480aacf727742 P 9d1c9b0b0124405398afc3033537cf76: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
W20260806 02:52:05.390275 29681 tablet.cc:2369] T 4b5f5594d84b471da53480aacf727742 P 9d1c9b0b0124405398afc3033537cf76: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:52:05.482407 29928 raft_consensus.cc:482] T 4b5f5594d84b471da53480aacf727742 P b5272675298b46c4b9bea5414bd422b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:52:05.482831 29928 raft_consensus.cc:504] T 4b5f5594d84b471da53480aacf727742 P b5272675298b46c4b9bea5414bd422b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5272675298b46c4b9bea5414bd422b4" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 33379 } } peers { permanent_uuid: "9d1c9b0b0124405398afc3033537cf76" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46561 } } peers { permanent_uuid: "d3114496d23749e48a2889e0a3ceef40" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45757 } }
I20260806 02:52:05.484660 29928 leader_election.cc:288] T 4b5f5594d84b471da53480aacf727742 P b5272675298b46c4b9bea5414bd422b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d1c9b0b0124405398afc3033537cf76 (127.25.211.130:46561), d3114496d23749e48a2889e0a3ceef40 (127.25.211.131:45757)
W20260806 02:52:05.487783 29888 tablet.cc:2369] T 4b5f5594d84b471da53480aacf727742 P d3114496d23749e48a2889e0a3ceef40: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:52:05.492969 29908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:05.485289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:45473 (local address 127.25.211.131:45757)
0806 02:52:05.485535 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:05.485543 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:05.486364 (+ 821us) server_negotiation.cc:407] Connection header received
0806 02:52:05.486534 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:05.486539 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:05.486619 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:05.486708 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:52:05.487385 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:05.488258 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:05.491220 (+ 2962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:05.492053 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:05.492197 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:05.492253 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:05.492561 (+ 308us) server_negotiation.cc:299] Negotiation successful
0806 02:52:05.492696 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260806 02:52:05.493085 29931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:05.485266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.131:45757 (local address 127.25.211.129:45473)
0806 02:52:05.486241 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:05.486260 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:05.486394 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:05.486756 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:05.486760 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:05.486785 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:52:05.487220 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:05.487230 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:05.488386 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:05.488391 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:05.491019 (+ 2628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:05.491038 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:05.492253 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:05.492258 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:05.492382 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:05.492408 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 02:52:05.492497 (+ 89us) client_negotiation.cc:240] Negotiation successful
0806 02:52:05.492694 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":820,"mutex_wait_us":66,"thread_start_us":538,"threads_started":1}
I20260806 02:52:05.494095 29911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:05.485264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:44671 (local address 127.25.211.130:46561)
0806 02:52:05.485490 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:05.485497 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:05.486263 (+ 766us) server_negotiation.cc:407] Connection header received
0806 02:52:05.486492 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:05.486497 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:05.486579 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:05.486666 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:52:05.487329 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:05.488355 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:05.492342 (+ 3987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:05.493156 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:05.493331 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:05.493403 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:05.493800 (+ 397us) server_negotiation.cc:299] Negotiation successful
0806 02:52:05.493907 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260806 02:52:05.494351 29930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:05.485171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.130:46561 (local address 127.25.211.129:44671)
0806 02:52:05.486112 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:05.486129 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:05.486341 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:05.486723 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:05.486727 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:05.486753 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:52:05.487161 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:05.487172 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:05.488500 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:05.488506 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:05.491019 (+ 2513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:05.491038 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:05.493395 (+ 2357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:05.493401 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:05.493529 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:05.493553 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 02:52:05.493785 (+ 232us) client_negotiation.cc:240] Negotiation successful
0806 02:52:05.493932 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":804,"mutex_wait_us":88,"thread_start_us":583,"threads_started":1}
I20260806 02:52:05.494259 29813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5f5594d84b471da53480aacf727742" candidate_uuid: "b5272675298b46c4b9bea5414bd422b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3114496d23749e48a2889e0a3ceef40" is_pre_election: true
I20260806 02:52:05.494853 29813 raft_consensus.cc:2454] T 4b5f5594d84b471da53480aacf727742 P d3114496d23749e48a2889e0a3ceef40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5272675298b46c4b9bea5414bd422b4 in term 0.
I20260806 02:52:05.495476 29613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5f5594d84b471da53480aacf727742" candidate_uuid: "b5272675298b46c4b9bea5414bd422b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d1c9b0b0124405398afc3033537cf76" is_pre_election: true
I20260806 02:52:05.495555 29300 leader_election.cc:302] T 4b5f5594d84b471da53480aacf727742 P b5272675298b46c4b9bea5414bd422b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b5272675298b46c4b9bea5414bd422b4, d3114496d23749e48a2889e0a3ceef40; no voters:
I20260806 02:52:05.495990 29928 raft_consensus.cc:2781] T 4b5f5594d84b471da53480aacf727742 P b5272675298b46c4b9bea5414bd422b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:52:05.496001 29613 raft_consensus.cc:2454] T 4b5f5594d84b471da53480aacf727742 P 9d1c9b0b0124405398afc3033537cf76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5272675298b46c4b9bea5414bd422b4 in term 0.
I20260806 02:52:05.496126 29928 raft_consensus.cc:482] T 4b5f5594d84b471da53480aacf727742 P b5272675298b46c4b9bea5414bd422b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:52:05.496217 29928 raft_consensus.cc:3037] T 4b5f5594d84b471da53480aacf727742 P b5272675298b46c4b9bea5414bd422b4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:05.498651 29928 raft_consensus.cc:504] T 4b5f5594d84b471da53480aacf727742 P b5272675298b46c4b9bea5414bd422b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5272675298b46c4b9bea5414bd422b4" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 33379 } } peers { permanent_uuid: "9d1c9b0b0124405398afc3033537cf76" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46561 } } peers { permanent_uuid: "d3114496d23749e48a2889e0a3ceef40" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45757 } }
I20260806 02:52:05.499310 29928 leader_election.cc:288] T 4b5f5594d84b471da53480aacf727742 P b5272675298b46c4b9bea5414bd422b4 [CANDIDATE]: Term 1 election: Requested vote from peers 9d1c9b0b0124405398afc3033537cf76 (127.25.211.130:46561), d3114496d23749e48a2889e0a3ceef40 (127.25.211.131:45757)
I20260806 02:52:05.499933 29613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5f5594d84b471da53480aacf727742" candidate_uuid: "b5272675298b46c4b9bea5414bd422b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d1c9b0b0124405398afc3033537cf76"
I20260806 02:52:05.499933 29813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5f5594d84b471da53480aacf727742" candidate_uuid: "b5272675298b46c4b9bea5414bd422b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3114496d23749e48a2889e0a3ceef40"
I20260806 02:52:05.500167 29613 raft_consensus.cc:3037] T 4b5f5594d84b471da53480aacf727742 P 9d1c9b0b0124405398afc3033537cf76 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:05.500167 29813 raft_consensus.cc:3037] T 4b5f5594d84b471da53480aacf727742 P d3114496d23749e48a2889e0a3ceef40 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:05.503415 29613 raft_consensus.cc:2454] T 4b5f5594d84b471da53480aacf727742 P 9d1c9b0b0124405398afc3033537cf76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5272675298b46c4b9bea5414bd422b4 in term 1.
I20260806 02:52:05.503415 29813 raft_consensus.cc:2454] T 4b5f5594d84b471da53480aacf727742 P d3114496d23749e48a2889e0a3ceef40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5272675298b46c4b9bea5414bd422b4 in term 1.
I20260806 02:52:05.503911 29301 leader_election.cc:302] T 4b5f5594d84b471da53480aacf727742 P b5272675298b46c4b9bea5414bd422b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d1c9b0b0124405398afc3033537cf76, b5272675298b46c4b9bea5414bd422b4; no voters:
I20260806 02:52:05.504257 29928 raft_consensus.cc:2781] T 4b5f5594d84b471da53480aacf727742 P b5272675298b46c4b9bea5414bd422b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:52:05.504736 29928 raft_consensus.cc:686] T 4b5f5594d84b471da53480aacf727742 P b5272675298b46c4b9bea5414bd422b4 [term 1 LEADER]: Becoming Leader. State: Replica: b5272675298b46c4b9bea5414bd422b4, State: Running, Role: LEADER
I20260806 02:52:05.505210 29928 consensus_queue.cc:237] T 4b5f5594d84b471da53480aacf727742 P b5272675298b46c4b9bea5414bd422b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5272675298b46c4b9bea5414bd422b4" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 33379 } } peers { permanent_uuid: "9d1c9b0b0124405398afc3033537cf76" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46561 } } peers { permanent_uuid: "d3114496d23749e48a2889e0a3ceef40" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45757 } }
I20260806 02:52:05.509475 29172 catalog_manager.cc:5495] T 4b5f5594d84b471da53480aacf727742 P b5272675298b46c4b9bea5414bd422b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to b5272675298b46c4b9bea5414bd422b4 (127.25.211.129). New cstate: current_term: 1 leader_uuid: "b5272675298b46c4b9bea5414bd422b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5272675298b46c4b9bea5414bd422b4" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 33379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d1c9b0b0124405398afc3033537cf76" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3114496d23749e48a2889e0a3ceef40" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45757 } health_report { overall_health: UNKNOWN } } }
Aug 06 02:52:05 dist-test-slave-zj8q krb5kdc[29106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984723, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.190@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:52:05.607612 29941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:05.592890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:38929 (local address 127.0.0.1:48578)
0806 02:52:05.593508 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:05.593529 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:05.593675 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:05.593990 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:05.593993 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:05.594855 (+ 862us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:05.595141 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:05.595153 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:05.596316 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:05.596320 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:05.597380 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:05.597394 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:05.598264 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:05.598269 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:05.598390 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:05.599121 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:52:05.599140 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:05.601593 (+ 2453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:05.604690 (+ 3097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:05.604707 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:05.604714 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:05.605194 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:05.605659 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:05.605663 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:05.605665 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:05.605821 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:05.606513 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:05.606590 (+ 77us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:05.606781 (+ 191us) client_negotiation.cc:769] Sending connection context
0806 02:52:05.606942 (+ 161us) client_negotiation.cc:240] Negotiation successful
0806 02:52:05.607247 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":369,"threads_started":1}
I20260806 02:52:05.607744 29719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:05.592866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48578 (local address 127.25.211.190:38929)
0806 02:52:05.593058 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:05.593068 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:05.593636 (+ 568us) server_negotiation.cc:407] Connection header received
0806 02:52:05.593774 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:05.593778 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:05.593846 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:05.593932 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:05.595341 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:05.596157 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:05.597537 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:05.598045 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:05.598187 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:05.601822 (+ 3635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:05.601828 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:05.601831 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:05.601871 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:05.604509 (+ 2638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:05.605364 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:05.605369 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:05.605372 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:05.605543 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:05.605956 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:05.605960 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:05.605962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:05.606364 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:05.606489 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:05.607106 (+ 617us) server_negotiation.cc:299] Negotiation successful
0806 02:52:05.607421 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":81,"server-negotiator.queue_time_us":70}
W20260806 02:52:05.610988 29477 tablet.cc:2369] T 4b5f5594d84b471da53480aacf727742 P b5272675298b46c4b9bea5414bd422b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:52:05.613246 29172 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:48578:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 02:52:05.614490 29172 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:52:05.622323 29577 tablet_service.cc:1467] Processing CreateTablet for tablet c641b49d0b19402bb73adc3615d4b8f0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3549713fe401427ea336918ef81af95d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 02:52:05.622213 29380 tablet_service.cc:1467] Processing CreateTablet for tablet c641b49d0b19402bb73adc3615d4b8f0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3549713fe401427ea336918ef81af95d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 02:52:05.622498 29782 tablet_service.cc:1467] Processing CreateTablet for tablet c641b49d0b19402bb73adc3615d4b8f0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3549713fe401427ea336918ef81af95d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 02:52:05.623231 29577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c641b49d0b19402bb73adc3615d4b8f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:05.623284 29380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c641b49d0b19402bb73adc3615d4b8f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:05.623385 29782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c641b49d0b19402bb73adc3615d4b8f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:05.629436 29921 tablet_bootstrap.cc:492] T c641b49d0b19402bb73adc3615d4b8f0 P 9d1c9b0b0124405398afc3033537cf76: Bootstrap starting.
I20260806 02:52:05.630863 29916 tablet_bootstrap.cc:492] T c641b49d0b19402bb73adc3615d4b8f0 P b5272675298b46c4b9bea5414bd422b4: Bootstrap starting.
I20260806 02:52:05.630858 29914 tablet_bootstrap.cc:492] T c641b49d0b19402bb73adc3615d4b8f0 P d3114496d23749e48a2889e0a3ceef40: Bootstrap starting.
I20260806 02:52:05.631063 29921 tablet_bootstrap.cc:654] T c641b49d0b19402bb73adc3615d4b8f0 P 9d1c9b0b0124405398afc3033537cf76: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:05.632543 29916 tablet_bootstrap.cc:654] T c641b49d0b19402bb73adc3615d4b8f0 P b5272675298b46c4b9bea5414bd422b4: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:05.632542 29914 tablet_bootstrap.cc:654] T c641b49d0b19402bb73adc3615d4b8f0 P d3114496d23749e48a2889e0a3ceef40: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:05.634356 29921 tablet_bootstrap.cc:492] T c641b49d0b19402bb73adc3615d4b8f0 P 9d1c9b0b0124405398afc3033537cf76: No bootstrap required, opened a new log
I20260806 02:52:05.634574 29921 ts_tablet_manager.cc:1397] T c641b49d0b19402bb73adc3615d4b8f0 P 9d1c9b0b0124405398afc3033537cf76: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 02:52:05.636029 29921 raft_consensus.cc:348] T c641b49d0b19402bb73adc3615d4b8f0 P 9d1c9b0b0124405398afc3033537cf76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5272675298b46c4b9bea5414bd422b4" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 33379 } } peers { permanent_uuid: "9d1c9b0b0124405398afc3033537cf76" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46561 } } peers { permanent_uuid: "d3114496d23749e48a2889e0a3ceef40" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45757 } }
I20260806 02:52:05.636618 29921 raft_consensus.cc:374] T c641b49d0b19402bb73adc3615d4b8f0 P 9d1c9b0b0124405398afc3033537cf76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:05.636736 29921 raft_consensus.cc:729] T c641b49d0b19402bb73adc3615d4b8f0 P 9d1c9b0b0124405398afc3033537cf76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d1c9b0b0124405398afc3033537cf76, State: Initialized, Role: FOLLOWER
I20260806 02:52:05.638453 29921 consensus_queue.cc:260] T c641b49d0b19402bb73adc3615d4b8f0 P 9d1c9b0b0124405398afc3033537cf76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5272675298b46c4b9bea5414bd422b4" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 33379 } } peers { permanent_uuid: "9d1c9b0b0124405398afc3033537cf76" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46561 } } peers { permanent_uuid: "d3114496d23749e48a2889e0a3ceef40" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45757 } }
I20260806 02:52:05.639060 29914 tablet_bootstrap.cc:492] T c641b49d0b19402bb73adc3615d4b8f0 P d3114496d23749e48a2889e0a3ceef40: No bootstrap required, opened a new log
I20260806 02:52:05.639194 29921 ts_tablet_manager.cc:1428] T c641b49d0b19402bb73adc3615d4b8f0 P 9d1c9b0b0124405398afc3033537cf76: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260806 02:52:05.639241 29914 ts_tablet_manager.cc:1397] T c641b49d0b19402bb73adc3615d4b8f0 P d3114496d23749e48a2889e0a3ceef40: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260806 02:52:05.639390 29916 tablet_bootstrap.cc:492] T c641b49d0b19402bb73adc3615d4b8f0 P b5272675298b46c4b9bea5414bd422b4: No bootstrap required, opened a new log
I20260806 02:52:05.639602 29916 ts_tablet_manager.cc:1397] T c641b49d0b19402bb73adc3615d4b8f0 P b5272675298b46c4b9bea5414bd422b4: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20260806 02:52:05.640254 29929 tablet_replica.cc:439] T c641b49d0b19402bb73adc3615d4b8f0 P 9d1c9b0b0124405398afc3033537cf76: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5272675298b46c4b9bea5414bd422b4" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 33379 } } peers { permanent_uuid: "9d1c9b0b0124405398afc3033537cf76" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46561 } } peers { permanent_uuid: "d3114496d23749e48a2889e0a3ceef40" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45757 } } }
I20260806 02:52:05.640686 29929 tablet_replica.cc:442] T c641b49d0b19402bb73adc3615d4b8f0 P 9d1c9b0b0124405398afc3033537cf76: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:52:05.640473 29916 raft_consensus.cc:348] T c641b49d0b19402bb73adc3615d4b8f0 P b5272675298b46c4b9bea5414bd422b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5272675298b46c4b9bea5414bd422b4" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 33379 } } peers { permanent_uuid: "9d1c9b0b0124405398afc3033537cf76" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46561 } } peers { permanent_uuid: "d3114496d23749e48a2889e0a3ceef40" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45757 } }
I20260806 02:52:05.641018 29914 raft_consensus.cc:348] T c641b49d0b19402bb73adc3615d4b8f0 P d3114496d23749e48a2889e0a3ceef40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5272675298b46c4b9bea5414bd422b4" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 33379 } } peers { permanent_uuid: "9d1c9b0b0124405398afc3033537cf76" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46561 } } peers { permanent_uuid: "d3114496d23749e48a2889e0a3ceef40" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45757 } }
I20260806 02:52:05.641218 29916 raft_consensus.cc:374] T c641b49d0b19402bb73adc3615d4b8f0 P b5272675298b46c4b9bea5414bd422b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:05.641309 29914 raft_consensus.cc:374] T c641b49d0b19402bb73adc3615d4b8f0 P d3114496d23749e48a2889e0a3ceef40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:05.641400 29914 raft_consensus.cc:729] T c641b49d0b19402bb73adc3615d4b8f0 P d3114496d23749e48a2889e0a3ceef40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3114496d23749e48a2889e0a3ceef40, State: Initialized, Role: FOLLOWER
I20260806 02:52:05.641441 29916 raft_consensus.cc:729] T c641b49d0b19402bb73adc3615d4b8f0 P b5272675298b46c4b9bea5414bd422b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5272675298b46c4b9bea5414bd422b4, State: Initialized, Role: FOLLOWER
I20260806 02:52:05.641804 29916 consensus_queue.cc:260] T c641b49d0b19402bb73adc3615d4b8f0 P b5272675298b46c4b9bea5414bd422b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5272675298b46c4b9bea5414bd422b4" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 33379 } } peers { permanent_uuid: "9d1c9b0b0124405398afc3033537cf76" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46561 } } peers { permanent_uuid: "d3114496d23749e48a2889e0a3ceef40" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45757 } }
I20260806 02:52:05.642333 29928 tablet_replica.cc:439] T c641b49d0b19402bb73adc3615d4b8f0 P b5272675298b46c4b9bea5414bd422b4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5272675298b46c4b9bea5414bd422b4" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 33379 } } peers { permanent_uuid: "9d1c9b0b0124405398afc3033537cf76" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46561 } } peers { permanent_uuid: "d3114496d23749e48a2889e0a3ceef40" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45757 } } }
I20260806 02:52:05.642648 29928 tablet_replica.cc:442] T c641b49d0b19402bb73adc3615d4b8f0 P b5272675298b46c4b9bea5414bd422b4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:52:05.643214 29916 ts_tablet_manager.cc:1428] T c641b49d0b19402bb73adc3615d4b8f0 P b5272675298b46c4b9bea5414bd422b4: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260806 02:52:05.641762 29914 consensus_queue.cc:260] T c641b49d0b19402bb73adc3615d4b8f0 P d3114496d23749e48a2889e0a3ceef40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5272675298b46c4b9bea5414bd422b4" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 33379 } } peers { permanent_uuid: "9d1c9b0b0124405398afc3033537cf76" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46561 } } peers { permanent_uuid: "d3114496d23749e48a2889e0a3ceef40" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45757 } }
I20260806 02:52:05.644299 29925 tablet_replica.cc:439] T c641b49d0b19402bb73adc3615d4b8f0 P d3114496d23749e48a2889e0a3ceef40: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5272675298b46c4b9bea5414bd422b4" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 33379 } } peers { permanent_uuid: "9d1c9b0b0124405398afc3033537cf76" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46561 } } peers { permanent_uuid: "d3114496d23749e48a2889e0a3ceef40" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45757 } } }
I20260806 02:52:05.644693 29925 tablet_replica.cc:442] T c641b49d0b19402bb73adc3615d4b8f0 P d3114496d23749e48a2889e0a3ceef40: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:52:05.645246 29914 ts_tablet_manager.cc:1428] T c641b49d0b19402bb73adc3615d4b8f0 P d3114496d23749e48a2889e0a3ceef40: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260806 02:52:05.834445 29928 raft_consensus.cc:482] T c641b49d0b19402bb73adc3615d4b8f0 P b5272675298b46c4b9bea5414bd422b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:52:05.834668 29928 raft_consensus.cc:504] T c641b49d0b19402bb73adc3615d4b8f0 P b5272675298b46c4b9bea5414bd422b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5272675298b46c4b9bea5414bd422b4" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 33379 } } peers { permanent_uuid: "9d1c9b0b0124405398afc3033537cf76" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46561 } } peers { permanent_uuid: "d3114496d23749e48a2889e0a3ceef40" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45757 } }
I20260806 02:52:05.835315 29928 leader_election.cc:288] T c641b49d0b19402bb73adc3615d4b8f0 P b5272675298b46c4b9bea5414bd422b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d1c9b0b0124405398afc3033537cf76 (127.25.211.130:46561), d3114496d23749e48a2889e0a3ceef40 (127.25.211.131:45757)
I20260806 02:52:05.835994 29813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c641b49d0b19402bb73adc3615d4b8f0" candidate_uuid: "b5272675298b46c4b9bea5414bd422b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3114496d23749e48a2889e0a3ceef40" is_pre_election: true
I20260806 02:52:05.836082 29613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c641b49d0b19402bb73adc3615d4b8f0" candidate_uuid: "b5272675298b46c4b9bea5414bd422b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d1c9b0b0124405398afc3033537cf76" is_pre_election: true
I20260806 02:52:05.836297 29813 raft_consensus.cc:2454] T c641b49d0b19402bb73adc3615d4b8f0 P d3114496d23749e48a2889e0a3ceef40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5272675298b46c4b9bea5414bd422b4 in term 0.
I20260806 02:52:05.836340 29613 raft_consensus.cc:2454] T c641b49d0b19402bb73adc3615d4b8f0 P 9d1c9b0b0124405398afc3033537cf76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5272675298b46c4b9bea5414bd422b4 in term 0.
I20260806 02:52:05.836807 29301 leader_election.cc:302] T c641b49d0b19402bb73adc3615d4b8f0 P b5272675298b46c4b9bea5414bd422b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d1c9b0b0124405398afc3033537cf76, b5272675298b46c4b9bea5414bd422b4; no voters:
I20260806 02:52:05.837121 29928 raft_consensus.cc:2781] T c641b49d0b19402bb73adc3615d4b8f0 P b5272675298b46c4b9bea5414bd422b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:52:05.837288 29928 raft_consensus.cc:482] T c641b49d0b19402bb73adc3615d4b8f0 P b5272675298b46c4b9bea5414bd422b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:52:05.837416 29928 raft_consensus.cc:3037] T c641b49d0b19402bb73adc3615d4b8f0 P b5272675298b46c4b9bea5414bd422b4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:05.839673 29928 raft_consensus.cc:504] T c641b49d0b19402bb73adc3615d4b8f0 P b5272675298b46c4b9bea5414bd422b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5272675298b46c4b9bea5414bd422b4" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 33379 } } peers { permanent_uuid: "9d1c9b0b0124405398afc3033537cf76" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46561 } } peers { permanent_uuid: "d3114496d23749e48a2889e0a3ceef40" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45757 } }
I20260806 02:52:05.840349 29928 leader_election.cc:288] T c641b49d0b19402bb73adc3615d4b8f0 P b5272675298b46c4b9bea5414bd422b4 [CANDIDATE]: Term 1 election: Requested vote from peers 9d1c9b0b0124405398afc3033537cf76 (127.25.211.130:46561), d3114496d23749e48a2889e0a3ceef40 (127.25.211.131:45757)
I20260806 02:52:05.841023 29613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c641b49d0b19402bb73adc3615d4b8f0" candidate_uuid: "b5272675298b46c4b9bea5414bd422b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d1c9b0b0124405398afc3033537cf76"
I20260806 02:52:05.841022 29813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c641b49d0b19402bb73adc3615d4b8f0" candidate_uuid: "b5272675298b46c4b9bea5414bd422b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3114496d23749e48a2889e0a3ceef40"
I20260806 02:52:05.841293 29813 raft_consensus.cc:3037] T c641b49d0b19402bb73adc3615d4b8f0 P d3114496d23749e48a2889e0a3ceef40 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:05.841295 29613 raft_consensus.cc:3037] T c641b49d0b19402bb73adc3615d4b8f0 P 9d1c9b0b0124405398afc3033537cf76 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:05.844264 29613 raft_consensus.cc:2454] T c641b49d0b19402bb73adc3615d4b8f0 P 9d1c9b0b0124405398afc3033537cf76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5272675298b46c4b9bea5414bd422b4 in term 1.
I20260806 02:52:05.844269 29813 raft_consensus.cc:2454] T c641b49d0b19402bb73adc3615d4b8f0 P d3114496d23749e48a2889e0a3ceef40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5272675298b46c4b9bea5414bd422b4 in term 1.
I20260806 02:52:05.844815 29300 leader_election.cc:302] T c641b49d0b19402bb73adc3615d4b8f0 P b5272675298b46c4b9bea5414bd422b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b5272675298b46c4b9bea5414bd422b4, d3114496d23749e48a2889e0a3ceef40; no voters:
I20260806 02:52:05.845142 29928 raft_consensus.cc:2781] T c641b49d0b19402bb73adc3615d4b8f0 P b5272675298b46c4b9bea5414bd422b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:52:05.845315 29928 raft_consensus.cc:686] T c641b49d0b19402bb73adc3615d4b8f0 P b5272675298b46c4b9bea5414bd422b4 [term 1 LEADER]: Becoming Leader. State: Replica: b5272675298b46c4b9bea5414bd422b4, State: Running, Role: LEADER
I20260806 02:52:05.845592 29928 consensus_queue.cc:237] T c641b49d0b19402bb73adc3615d4b8f0 P b5272675298b46c4b9bea5414bd422b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5272675298b46c4b9bea5414bd422b4" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 33379 } } peers { permanent_uuid: "9d1c9b0b0124405398afc3033537cf76" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46561 } } peers { permanent_uuid: "d3114496d23749e48a2889e0a3ceef40" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45757 } }
I20260806 02:52:05.847067 29932 tablet_replica.cc:439] T c641b49d0b19402bb73adc3615d4b8f0 P b5272675298b46c4b9bea5414bd422b4: TxnStatusTablet state changed. Reason: New leader b5272675298b46c4b9bea5414bd422b4. Latest consensus state: current_term: 1 leader_uuid: "b5272675298b46c4b9bea5414bd422b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5272675298b46c4b9bea5414bd422b4" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 33379 } } peers { permanent_uuid: "9d1c9b0b0124405398afc3033537cf76" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46561 } } peers { permanent_uuid: "d3114496d23749e48a2889e0a3ceef40" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45757 } } }
I20260806 02:52:05.847311 29932 tablet_replica.cc:442] T c641b49d0b19402bb73adc3615d4b8f0 P b5272675298b46c4b9bea5414bd422b4: This TxnStatusTablet replica's current role is: LEADER
I20260806 02:52:05.848083 29948 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 02:52:05.848361 29948 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 02:52:05.848691 29170 catalog_manager.cc:5495] T c641b49d0b19402bb73adc3615d4b8f0 P b5272675298b46c4b9bea5414bd422b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to b5272675298b46c4b9bea5414bd422b4 (127.25.211.129). New cstate: current_term: 1 leader_uuid: "b5272675298b46c4b9bea5414bd422b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5272675298b46c4b9bea5414bd422b4" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 33379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d1c9b0b0124405398afc3033537cf76" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3114496d23749e48a2889e0a3ceef40" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45757 } health_report { overall_health: UNKNOWN } } }
I20260806 02:52:05.879122 29941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:05.872070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:33379 (local address 127.0.0.1:52306)
0806 02:52:05.872343 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:05.872367 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:05.872534 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:05.873207 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:05.873211 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:05.873221 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:52:05.873378 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:05.873386 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:05.875020 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:05.875025 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:05.876033 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:05.876048 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:05.877102 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:05.877108 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:05.877301 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:05.877395 (+ 94us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:52:05.878643 (+ 1248us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:52:05.878668 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 02:52:05.878739 (+ 71us) client_negotiation.cc:240] Negotiation successful
0806 02:52:05.878871 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260806 02:52:05.879477 29956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:05.872168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52306 (local address 127.25.211.129:33379)
0806 02:52:05.872767 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:05.872773 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:05.872794 (+ 21us) server_negotiation.cc:407] Connection header received
0806 02:52:05.872892 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:05.872897 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:05.873006 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:05.873185 (+ 179us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:52:05.873523 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:05.874880 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:05.876191 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:05.876900 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:05.877030 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:05.877646 (+ 616us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:52:05.878531 (+ 885us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:52:05.878683 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:05.878783 (+ 100us) server_negotiation.cc:299] Negotiation successful
0806 02:52:05.878874 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":289,"threads_started":1}
I20260806 02:52:05.887742 29813 raft_consensus.cc:1261] T c641b49d0b19402bb73adc3615d4b8f0 P d3114496d23749e48a2889e0a3ceef40 [term 1 FOLLOWER]: Refusing update from remote peer b5272675298b46c4b9bea5414bd422b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:52:05.887791 29613 raft_consensus.cc:1261] T c641b49d0b19402bb73adc3615d4b8f0 P 9d1c9b0b0124405398afc3033537cf76 [term 1 FOLLOWER]: Refusing update from remote peer b5272675298b46c4b9bea5414bd422b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:52:05.888727 29928 consensus_queue.cc:1035] T c641b49d0b19402bb73adc3615d4b8f0 P b5272675298b46c4b9bea5414bd422b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3114496d23749e48a2889e0a3ceef40" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:52:05.889343 29928 consensus_queue.cc:1035] T c641b49d0b19402bb73adc3615d4b8f0 P b5272675298b46c4b9bea5414bd422b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d1c9b0b0124405398afc3033537cf76" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:52:05.897225 29925 tablet_replica.cc:439] T c641b49d0b19402bb73adc3615d4b8f0 P d3114496d23749e48a2889e0a3ceef40: TxnStatusTablet state changed. Reason: New leader b5272675298b46c4b9bea5414bd422b4. Latest consensus state: current_term: 1 leader_uuid: "b5272675298b46c4b9bea5414bd422b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5272675298b46c4b9bea5414bd422b4" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 33379 } } peers { permanent_uuid: "9d1c9b0b0124405398afc3033537cf76" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46561 } } peers { permanent_uuid: "d3114496d23749e48a2889e0a3ceef40" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45757 } } }
I20260806 02:52:05.897528 29925 tablet_replica.cc:442] T c641b49d0b19402bb73adc3615d4b8f0 P d3114496d23749e48a2889e0a3ceef40: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:52:05.900359 29958 mvcc.cc:205] Tried to move back new op lower bound from 7315393437225824256 to 7315393437066063872. Current Snapshot: MvccSnapshot[applied={T|T < 7315393437225824256}]
I20260806 02:52:05.901577 29932 tablet_replica.cc:439] T c641b49d0b19402bb73adc3615d4b8f0 P b5272675298b46c4b9bea5414bd422b4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b5272675298b46c4b9bea5414bd422b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5272675298b46c4b9bea5414bd422b4" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 33379 } } peers { permanent_uuid: "9d1c9b0b0124405398afc3033537cf76" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46561 } } peers { permanent_uuid: "d3114496d23749e48a2889e0a3ceef40" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45757 } } }
I20260806 02:52:05.901898 29932 tablet_replica.cc:442] T c641b49d0b19402bb73adc3615d4b8f0 P b5272675298b46c4b9bea5414bd422b4: This TxnStatusTablet replica's current role is: LEADER
I20260806 02:52:05.908052 29925 tablet_replica.cc:439] T c641b49d0b19402bb73adc3615d4b8f0 P d3114496d23749e48a2889e0a3ceef40: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b5272675298b46c4b9bea5414bd422b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5272675298b46c4b9bea5414bd422b4" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 33379 } } peers { permanent_uuid: "9d1c9b0b0124405398afc3033537cf76" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46561 } } peers { permanent_uuid: "d3114496d23749e48a2889e0a3ceef40" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45757 } } }
I20260806 02:52:05.908329 29925 tablet_replica.cc:442] T c641b49d0b19402bb73adc3615d4b8f0 P d3114496d23749e48a2889e0a3ceef40: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:52:05.915093 29929 tablet_replica.cc:439] T c641b49d0b19402bb73adc3615d4b8f0 P 9d1c9b0b0124405398afc3033537cf76: TxnStatusTablet state changed. Reason: New leader b5272675298b46c4b9bea5414bd422b4. Latest consensus state: current_term: 1 leader_uuid: "b5272675298b46c4b9bea5414bd422b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5272675298b46c4b9bea5414bd422b4" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 33379 } } peers { permanent_uuid: "9d1c9b0b0124405398afc3033537cf76" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46561 } } peers { permanent_uuid: "d3114496d23749e48a2889e0a3ceef40" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45757 } } }
I20260806 02:52:05.915448 29929 tablet_replica.cc:442] T c641b49d0b19402bb73adc3615d4b8f0 P 9d1c9b0b0124405398afc3033537cf76: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:52:05.918965 29932 tablet_replica.cc:439] T c641b49d0b19402bb73adc3615d4b8f0 P b5272675298b46c4b9bea5414bd422b4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b5272675298b46c4b9bea5414bd422b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5272675298b46c4b9bea5414bd422b4" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 33379 } } peers { permanent_uuid: "9d1c9b0b0124405398afc3033537cf76" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46561 } } peers { permanent_uuid: "d3114496d23749e48a2889e0a3ceef40" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45757 } } }
I20260806 02:52:05.919296 29932 tablet_replica.cc:442] T c641b49d0b19402bb73adc3615d4b8f0 P b5272675298b46c4b9bea5414bd422b4: This TxnStatusTablet replica's current role is: LEADER
I20260806 02:52:05.924203 29929 tablet_replica.cc:439] T c641b49d0b19402bb73adc3615d4b8f0 P 9d1c9b0b0124405398afc3033537cf76: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b5272675298b46c4b9bea5414bd422b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5272675298b46c4b9bea5414bd422b4" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 33379 } } peers { permanent_uuid: "9d1c9b0b0124405398afc3033537cf76" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46561 } } peers { permanent_uuid: "d3114496d23749e48a2889e0a3ceef40" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45757 } } }
I20260806 02:52:05.924484 29929 tablet_replica.cc:442] T c641b49d0b19402bb73adc3615d4b8f0 P 9d1c9b0b0124405398afc3033537cf76: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:52:05.926504 29956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:05.918425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52308 (local address 127.25.211.129:33379)
0806 02:52:05.918587 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:05.918593 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:05.920116 (+ 1523us) server_negotiation.cc:407] Connection header received
0806 02:52:05.920352 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:05.920359 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:05.920444 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:05.920511 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:52:05.921262 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:05.922416 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:05.924417 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:05.925064 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:05.925181 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:05.925750 (+ 569us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:52:05.925857 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:52:05.925938 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:05.926076 (+ 138us) server_negotiation.cc:299] Negotiation successful
0806 02:52:05.926181 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260806 02:52:05.945222 29956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:05.939194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:53599 (local address 127.25.211.129:33379)
0806 02:52:05.939408 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:05.939415 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:05.940140 (+ 725us) server_negotiation.cc:407] Connection header received
0806 02:52:05.940323 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:05.940328 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:05.940412 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:05.940488 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:52:05.941026 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:05.941805 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:05.943300 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:05.943765 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:05.943869 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:05.944347 (+ 478us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:52:05.944453 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:52:05.944560 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:05.944690 (+ 130us) server_negotiation.cc:299] Negotiation successful
0806 02:52:05.944828 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":59}
I20260806 02:52:05.945328 29976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:05.939245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:33379 (local address 127.25.211.129:53599)
0806 02:52:05.940033 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:05.940050 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:05.940205 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:05.940511 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:05.940515 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:05.940525 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:52:05.940874 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:05.940883 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:05.941938 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:05.941946 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:05.943166 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:05.943179 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:05.943944 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:05.943949 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:05.944087 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:05.944149 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:52:05.944572 (+ 423us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:52:05.944583 (+ 11us) client_negotiation.cc:769] Sending connection context
0806 02:52:05.944661 (+ 78us) client_negotiation.cc:240] Negotiation successful
0806 02:52:05.944902 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"mutex_wait_us":60,"thread_start_us":458,"threads_started":1}
I20260806 02:52:05.956637 29813 raft_consensus.cc:1261] T 4b5f5594d84b471da53480aacf727742 P d3114496d23749e48a2889e0a3ceef40 [term 1 FOLLOWER]: Refusing update from remote peer b5272675298b46c4b9bea5414bd422b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:52:05.957299 29613 raft_consensus.cc:1261] T 4b5f5594d84b471da53480aacf727742 P 9d1c9b0b0124405398afc3033537cf76 [term 1 FOLLOWER]: Refusing update from remote peer b5272675298b46c4b9bea5414bd422b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:52:05.957274 29928 consensus_queue.cc:1035] T 4b5f5594d84b471da53480aacf727742 P b5272675298b46c4b9bea5414bd422b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3114496d23749e48a2889e0a3ceef40" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:52:05.957965 29928 consensus_queue.cc:1035] T 4b5f5594d84b471da53480aacf727742 P b5272675298b46c4b9bea5414bd422b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d1c9b0b0124405398afc3033537cf76" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:52:05.965875 29959 mvcc.cc:205] Tried to move back new op lower bound from 7315393437512966144 to 7315393435673493504. Current Snapshot: MvccSnapshot[applied={T|T < 7315393437512966144}]
I20260806 02:52:06.033159 29170 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:52:06.033465 29170 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:52:06.037173 29170 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 50d796ee82c3472496eacc2910ede519: Sending DeleteTablet for 3 replicas of tablet 4b5f5594d84b471da53480aacf727742
I20260806 02:52:06.038409 29379 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b5f5594d84b471da53480aacf727742 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:52:06 UTC) from {username='kudu', principal='kudu/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:52294
I20260806 02:52:06.038748 29782 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b5f5594d84b471da53480aacf727742 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:52:06 UTC) from {username='kudu', principal='kudu/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:60664
I20260806 02:52:06.039637 29577 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b5f5594d84b471da53480aacf727742 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:52:06 UTC) from {username='kudu', principal='kudu/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:41152
I20260806 02:52:06.039809 29170 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48576:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:52:06.040512 29170 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:52:06.048956 29380 tablet_service.cc:1467] Processing CreateTablet for tablet e4eaa94c570047bc921189a3b81e1525 (DEFAULT_TABLE table=test-table [id=8cfe894039f3460381c6e51341be8de5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:52:06.049484 29380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4eaa94c570047bc921189a3b81e1525. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:06.050411 29986 tablet_replica.cc:330] T 4b5f5594d84b471da53480aacf727742 P 9d1c9b0b0124405398afc3033537cf76: stopping tablet replica
I20260806 02:52:06.050936 29986 raft_consensus.cc:2229] T 4b5f5594d84b471da53480aacf727742 P 9d1c9b0b0124405398afc3033537cf76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:52:06.051285 29986 raft_consensus.cc:2258] T 4b5f5594d84b471da53480aacf727742 P 9d1c9b0b0124405398afc3033537cf76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:52:06.052804 29984 tablet_replica.cc:330] T 4b5f5594d84b471da53480aacf727742 P b5272675298b46c4b9bea5414bd422b4: stopping tablet replica
I20260806 02:52:06.053273 29984 raft_consensus.cc:2229] T 4b5f5594d84b471da53480aacf727742 P b5272675298b46c4b9bea5414bd422b4 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:52:06.053700 29986 ts_tablet_manager.cc:1905] T 4b5f5594d84b471da53480aacf727742 P 9d1c9b0b0124405398afc3033537cf76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:52:06.053958 29984 raft_consensus.cc:2258] T 4b5f5594d84b471da53480aacf727742 P b5272675298b46c4b9bea5414bd422b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:52:06.055263 29781 tablet_service.cc:1467] Processing CreateTablet for tablet e4eaa94c570047bc921189a3b81e1525 (DEFAULT_TABLE table=test-table [id=8cfe894039f3460381c6e51341be8de5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:52:06.055809 29781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4eaa94c570047bc921189a3b81e1525. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:06.056028 29984 ts_tablet_manager.cc:1905] T 4b5f5594d84b471da53480aacf727742 P b5272675298b46c4b9bea5414bd422b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:52:06.058632 29576 tablet_service.cc:1467] Processing CreateTablet for tablet e4eaa94c570047bc921189a3b81e1525 (DEFAULT_TABLE table=test-table [id=8cfe894039f3460381c6e51341be8de5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:52:06.059279 29576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4eaa94c570047bc921189a3b81e1525. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:06.064358 29984 ts_tablet_manager.cc:1918] T 4b5f5594d84b471da53480aacf727742 P b5272675298b46c4b9bea5414bd422b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:52:06.064579 29984 log.cc:1196] T 4b5f5594d84b471da53480aacf727742 P b5272675298b46c4b9bea5414bd422b4: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/ts-0/wal/wals/4b5f5594d84b471da53480aacf727742
I20260806 02:52:06.065186 29984 ts_tablet_manager.cc:1939] T 4b5f5594d84b471da53480aacf727742 P b5272675298b46c4b9bea5414bd422b4: Deleting consensus metadata
I20260806 02:52:06.066540 29916 tablet_bootstrap.cc:492] T e4eaa94c570047bc921189a3b81e1525 P b5272675298b46c4b9bea5414bd422b4: Bootstrap starting.
I20260806 02:52:06.067246 29151 catalog_manager.cc:4841] TS b5272675298b46c4b9bea5414bd422b4 (127.25.211.129:33379): tablet 4b5f5594d84b471da53480aacf727742 (table test-table [id=756b4455f70943cdbbece77adce94357]) successfully deleted
I20260806 02:52:06.068411 29916 tablet_bootstrap.cc:654] T e4eaa94c570047bc921189a3b81e1525 P b5272675298b46c4b9bea5414bd422b4: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:06.068966 29914 tablet_bootstrap.cc:492] T e4eaa94c570047bc921189a3b81e1525 P d3114496d23749e48a2889e0a3ceef40: Bootstrap starting.
I20260806 02:52:06.069794 29921 tablet_bootstrap.cc:492] T e4eaa94c570047bc921189a3b81e1525 P 9d1c9b0b0124405398afc3033537cf76: Bootstrap starting.
I20260806 02:52:06.070842 29914 tablet_bootstrap.cc:654] T e4eaa94c570047bc921189a3b81e1525 P d3114496d23749e48a2889e0a3ceef40: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:06.071489 29921 tablet_bootstrap.cc:654] T e4eaa94c570047bc921189a3b81e1525 P 9d1c9b0b0124405398afc3033537cf76: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:06.077329 29916 tablet_bootstrap.cc:492] T e4eaa94c570047bc921189a3b81e1525 P b5272675298b46c4b9bea5414bd422b4: No bootstrap required, opened a new log
I20260806 02:52:06.077657 29916 ts_tablet_manager.cc:1397] T e4eaa94c570047bc921189a3b81e1525 P b5272675298b46c4b9bea5414bd422b4: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.001s
I20260806 02:52:06.078656 29916 raft_consensus.cc:348] T e4eaa94c570047bc921189a3b81e1525 P b5272675298b46c4b9bea5414bd422b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5272675298b46c4b9bea5414bd422b4" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 33379 } } peers { permanent_uuid: "9d1c9b0b0124405398afc3033537cf76" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46561 } } peers { permanent_uuid: "d3114496d23749e48a2889e0a3ceef40" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45757 } }
I20260806 02:52:06.079061 29916 raft_consensus.cc:374] T e4eaa94c570047bc921189a3b81e1525 P b5272675298b46c4b9bea5414bd422b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:06.079196 29916 raft_consensus.cc:729] T e4eaa94c570047bc921189a3b81e1525 P b5272675298b46c4b9bea5414bd422b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5272675298b46c4b9bea5414bd422b4, State: Initialized, Role: FOLLOWER
I20260806 02:52:06.079553 29916 consensus_queue.cc:260] T e4eaa94c570047bc921189a3b81e1525 P b5272675298b46c4b9bea5414bd422b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5272675298b46c4b9bea5414bd422b4" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 33379 } } peers { permanent_uuid: "9d1c9b0b0124405398afc3033537cf76" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46561 } } peers { permanent_uuid: "d3114496d23749e48a2889e0a3ceef40" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45757 } }
I20260806 02:52:06.081954 29914 tablet_bootstrap.cc:492] T e4eaa94c570047bc921189a3b81e1525 P d3114496d23749e48a2889e0a3ceef40: No bootstrap required, opened a new log
I20260806 02:52:06.082134 29914 ts_tablet_manager.cc:1397] T e4eaa94c570047bc921189a3b81e1525 P d3114496d23749e48a2889e0a3ceef40: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.006s
I20260806 02:52:06.082978 29914 raft_consensus.cc:348] T e4eaa94c570047bc921189a3b81e1525 P d3114496d23749e48a2889e0a3ceef40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5272675298b46c4b9bea5414bd422b4" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 33379 } } peers { permanent_uuid: "9d1c9b0b0124405398afc3033537cf76" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46561 } } peers { permanent_uuid: "d3114496d23749e48a2889e0a3ceef40" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45757 } }
I20260806 02:52:06.083254 29914 raft_consensus.cc:374] T e4eaa94c570047bc921189a3b81e1525 P d3114496d23749e48a2889e0a3ceef40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:06.083344 29914 raft_consensus.cc:729] T e4eaa94c570047bc921189a3b81e1525 P d3114496d23749e48a2889e0a3ceef40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3114496d23749e48a2889e0a3ceef40, State: Initialized, Role: FOLLOWER
I20260806 02:52:06.083639 29914 consensus_queue.cc:260] T e4eaa94c570047bc921189a3b81e1525 P d3114496d23749e48a2889e0a3ceef40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5272675298b46c4b9bea5414bd422b4" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 33379 } } peers { permanent_uuid: "9d1c9b0b0124405398afc3033537cf76" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46561 } } peers { permanent_uuid: "d3114496d23749e48a2889e0a3ceef40" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45757 } }
I20260806 02:52:06.084483 29916 ts_tablet_manager.cc:1428] T e4eaa94c570047bc921189a3b81e1525 P b5272675298b46c4b9bea5414bd422b4: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260806 02:52:06.084921 29986 ts_tablet_manager.cc:1918] T 4b5f5594d84b471da53480aacf727742 P 9d1c9b0b0124405398afc3033537cf76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:52:06.085140 29986 log.cc:1196] T 4b5f5594d84b471da53480aacf727742 P 9d1c9b0b0124405398afc3033537cf76: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/ts-1/wal/wals/4b5f5594d84b471da53480aacf727742
I20260806 02:52:06.085784 29986 ts_tablet_manager.cc:1939] T 4b5f5594d84b471da53480aacf727742 P 9d1c9b0b0124405398afc3033537cf76: Deleting consensus metadata
I20260806 02:52:06.086915 29914 ts_tablet_manager.cc:1428] T e4eaa94c570047bc921189a3b81e1525 P d3114496d23749e48a2889e0a3ceef40: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260806 02:52:06.087983 29151 catalog_manager.cc:4841] TS 9d1c9b0b0124405398afc3033537cf76 (127.25.211.130:46561): tablet 4b5f5594d84b471da53480aacf727742 (table test-table [id=756b4455f70943cdbbece77adce94357]) successfully deleted
I20260806 02:52:06.089742 29921 tablet_bootstrap.cc:492] T e4eaa94c570047bc921189a3b81e1525 P 9d1c9b0b0124405398afc3033537cf76: No bootstrap required, opened a new log
I20260806 02:52:06.089931 29921 ts_tablet_manager.cc:1397] T e4eaa94c570047bc921189a3b81e1525 P 9d1c9b0b0124405398afc3033537cf76: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.005s
I20260806 02:52:06.090091 29985 tablet_replica.cc:330] T 4b5f5594d84b471da53480aacf727742 P d3114496d23749e48a2889e0a3ceef40: stopping tablet replica
I20260806 02:52:06.090615 29985 raft_consensus.cc:2229] T 4b5f5594d84b471da53480aacf727742 P d3114496d23749e48a2889e0a3ceef40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:52:06.090811 29921 raft_consensus.cc:348] T e4eaa94c570047bc921189a3b81e1525 P 9d1c9b0b0124405398afc3033537cf76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5272675298b46c4b9bea5414bd422b4" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 33379 } } peers { permanent_uuid: "9d1c9b0b0124405398afc3033537cf76" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46561 } } peers { permanent_uuid: "d3114496d23749e48a2889e0a3ceef40" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45757 } }
I20260806 02:52:06.091003 29985 raft_consensus.cc:2258] T 4b5f5594d84b471da53480aacf727742 P d3114496d23749e48a2889e0a3ceef40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:52:06.091049 29921 raft_consensus.cc:374] T e4eaa94c570047bc921189a3b81e1525 P 9d1c9b0b0124405398afc3033537cf76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:06.091116 29921 raft_consensus.cc:729] T e4eaa94c570047bc921189a3b81e1525 P 9d1c9b0b0124405398afc3033537cf76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d1c9b0b0124405398afc3033537cf76, State: Initialized, Role: FOLLOWER
I20260806 02:52:06.091358 29921 consensus_queue.cc:260] T e4eaa94c570047bc921189a3b81e1525 P 9d1c9b0b0124405398afc3033537cf76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5272675298b46c4b9bea5414bd422b4" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 33379 } } peers { permanent_uuid: "9d1c9b0b0124405398afc3033537cf76" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46561 } } peers { permanent_uuid: "d3114496d23749e48a2889e0a3ceef40" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45757 } }
I20260806 02:52:06.091940 29921 ts_tablet_manager.cc:1428] T e4eaa94c570047bc921189a3b81e1525 P 9d1c9b0b0124405398afc3033537cf76: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260806 02:52:06.093031 29985 ts_tablet_manager.cc:1905] T 4b5f5594d84b471da53480aacf727742 P d3114496d23749e48a2889e0a3ceef40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:52:06.099046 29985 ts_tablet_manager.cc:1918] T 4b5f5594d84b471da53480aacf727742 P d3114496d23749e48a2889e0a3ceef40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:52:06.099251 29985 log.cc:1196] T 4b5f5594d84b471da53480aacf727742 P d3114496d23749e48a2889e0a3ceef40: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/ts-2/wal/wals/4b5f5594d84b471da53480aacf727742
I20260806 02:52:06.099709 29985 ts_tablet_manager.cc:1939] T 4b5f5594d84b471da53480aacf727742 P d3114496d23749e48a2889e0a3ceef40: Deleting consensus metadata
I20260806 02:52:06.101711 29150 catalog_manager.cc:4841] TS d3114496d23749e48a2889e0a3ceef40 (127.25.211.131:45757): tablet 4b5f5594d84b471da53480aacf727742 (table test-table [id=756b4455f70943cdbbece77adce94357]) successfully deleted
I20260806 02:52:06.136332 29932 raft_consensus.cc:482] T e4eaa94c570047bc921189a3b81e1525 P b5272675298b46c4b9bea5414bd422b4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:52:06.136638 29932 raft_consensus.cc:504] T e4eaa94c570047bc921189a3b81e1525 P b5272675298b46c4b9bea5414bd422b4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5272675298b46c4b9bea5414bd422b4" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 33379 } } peers { permanent_uuid: "9d1c9b0b0124405398afc3033537cf76" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46561 } } peers { permanent_uuid: "d3114496d23749e48a2889e0a3ceef40" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45757 } }
I20260806 02:52:06.137445 29932 leader_election.cc:288] T e4eaa94c570047bc921189a3b81e1525 P b5272675298b46c4b9bea5414bd422b4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d1c9b0b0124405398afc3033537cf76 (127.25.211.130:46561), d3114496d23749e48a2889e0a3ceef40 (127.25.211.131:45757)
I20260806 02:52:06.138111 29813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4eaa94c570047bc921189a3b81e1525" candidate_uuid: "b5272675298b46c4b9bea5414bd422b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3114496d23749e48a2889e0a3ceef40" is_pre_election: true
I20260806 02:52:06.138129 29613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4eaa94c570047bc921189a3b81e1525" candidate_uuid: "b5272675298b46c4b9bea5414bd422b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d1c9b0b0124405398afc3033537cf76" is_pre_election: true
I20260806 02:52:06.138393 29813 raft_consensus.cc:2454] T e4eaa94c570047bc921189a3b81e1525 P d3114496d23749e48a2889e0a3ceef40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5272675298b46c4b9bea5414bd422b4 in term 0.
I20260806 02:52:06.138415 29613 raft_consensus.cc:2454] T e4eaa94c570047bc921189a3b81e1525 P 9d1c9b0b0124405398afc3033537cf76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5272675298b46c4b9bea5414bd422b4 in term 0.
I20260806 02:52:06.138952 29300 leader_election.cc:302] T e4eaa94c570047bc921189a3b81e1525 P b5272675298b46c4b9bea5414bd422b4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 9d1c9b0b0124405398afc3033537cf76, b5272675298b46c4b9bea5414bd422b4, d3114496d23749e48a2889e0a3ceef40; no voters:
I20260806 02:52:06.139310 29932 raft_consensus.cc:2781] T e4eaa94c570047bc921189a3b81e1525 P b5272675298b46c4b9bea5414bd422b4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:52:06.139442 29932 raft_consensus.cc:482] T e4eaa94c570047bc921189a3b81e1525 P b5272675298b46c4b9bea5414bd422b4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:52:06.139540 29932 raft_consensus.cc:3037] T e4eaa94c570047bc921189a3b81e1525 P b5272675298b46c4b9bea5414bd422b4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:06.142918 29932 raft_consensus.cc:504] T e4eaa94c570047bc921189a3b81e1525 P b5272675298b46c4b9bea5414bd422b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5272675298b46c4b9bea5414bd422b4" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 33379 } } peers { permanent_uuid: "9d1c9b0b0124405398afc3033537cf76" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46561 } } peers { permanent_uuid: "d3114496d23749e48a2889e0a3ceef40" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45757 } }
I20260806 02:52:06.143707 29932 leader_election.cc:288] T e4eaa94c570047bc921189a3b81e1525 P b5272675298b46c4b9bea5414bd422b4 [CANDIDATE]: Term 1 election: Requested vote from peers 9d1c9b0b0124405398afc3033537cf76 (127.25.211.130:46561), d3114496d23749e48a2889e0a3ceef40 (127.25.211.131:45757)
I20260806 02:52:06.144289 29613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4eaa94c570047bc921189a3b81e1525" candidate_uuid: "b5272675298b46c4b9bea5414bd422b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d1c9b0b0124405398afc3033537cf76"
I20260806 02:52:06.144330 29813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4eaa94c570047bc921189a3b81e1525" candidate_uuid: "b5272675298b46c4b9bea5414bd422b4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3114496d23749e48a2889e0a3ceef40"
I20260806 02:52:06.144497 29613 raft_consensus.cc:3037] T e4eaa94c570047bc921189a3b81e1525 P 9d1c9b0b0124405398afc3033537cf76 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:06.144567 29813 raft_consensus.cc:3037] T e4eaa94c570047bc921189a3b81e1525 P d3114496d23749e48a2889e0a3ceef40 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:06.146799 29613 raft_consensus.cc:2454] T e4eaa94c570047bc921189a3b81e1525 P 9d1c9b0b0124405398afc3033537cf76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5272675298b46c4b9bea5414bd422b4 in term 1.
I20260806 02:52:06.147284 29301 leader_election.cc:302] T e4eaa94c570047bc921189a3b81e1525 P b5272675298b46c4b9bea5414bd422b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d1c9b0b0124405398afc3033537cf76, b5272675298b46c4b9bea5414bd422b4; no voters:
I20260806 02:52:06.147518 29813 raft_consensus.cc:2454] T e4eaa94c570047bc921189a3b81e1525 P d3114496d23749e48a2889e0a3ceef40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5272675298b46c4b9bea5414bd422b4 in term 1.
I20260806 02:52:06.147589 29932 raft_consensus.cc:2781] T e4eaa94c570047bc921189a3b81e1525 P b5272675298b46c4b9bea5414bd422b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:52:06.147744 29932 raft_consensus.cc:686] T e4eaa94c570047bc921189a3b81e1525 P b5272675298b46c4b9bea5414bd422b4 [term 1 LEADER]: Becoming Leader. State: Replica: b5272675298b46c4b9bea5414bd422b4, State: Running, Role: LEADER
I20260806 02:52:06.148092 29932 consensus_queue.cc:237] T e4eaa94c570047bc921189a3b81e1525 P b5272675298b46c4b9bea5414bd422b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5272675298b46c4b9bea5414bd422b4" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 33379 } } peers { permanent_uuid: "9d1c9b0b0124405398afc3033537cf76" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46561 } } peers { permanent_uuid: "d3114496d23749e48a2889e0a3ceef40" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45757 } }
I20260806 02:52:06.151058 29170 catalog_manager.cc:5495] T e4eaa94c570047bc921189a3b81e1525 P b5272675298b46c4b9bea5414bd422b4 reported cstate change: term changed from 0 to 1, leader changed from <none> to b5272675298b46c4b9bea5414bd422b4 (127.25.211.129). New cstate: current_term: 1 leader_uuid: "b5272675298b46c4b9bea5414bd422b4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5272675298b46c4b9bea5414bd422b4" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 33379 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9d1c9b0b0124405398afc3033537cf76" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3114496d23749e48a2889e0a3ceef40" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45757 } health_report { overall_health: UNKNOWN } } }
I20260806 02:52:06.185539 29812 raft_consensus.cc:1261] T e4eaa94c570047bc921189a3b81e1525 P d3114496d23749e48a2889e0a3ceef40 [term 1 FOLLOWER]: Refusing update from remote peer b5272675298b46c4b9bea5414bd422b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:52:06.186405 29613 raft_consensus.cc:1261] T e4eaa94c570047bc921189a3b81e1525 P 9d1c9b0b0124405398afc3033537cf76 [term 1 FOLLOWER]: Refusing update from remote peer b5272675298b46c4b9bea5414bd422b4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:52:06.186311 29928 consensus_queue.cc:1035] T e4eaa94c570047bc921189a3b81e1525 P b5272675298b46c4b9bea5414bd422b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3114496d23749e48a2889e0a3ceef40" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:52:06.186971 29977 consensus_queue.cc:1035] T e4eaa94c570047bc921189a3b81e1525 P b5272675298b46c4b9bea5414bd422b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d1c9b0b0124405398afc3033537cf76" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 02:52:06.211012 29958 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260806 02:52:07.820212 29170 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:52:07.820474 29170 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:52:07.825316 29170 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 50d796ee82c3472496eacc2910ede519: Sending DeleteTablet for 3 replicas of tablet e4eaa94c570047bc921189a3b81e1525
I20260806 02:52:07.826356 29380 tablet_service.cc:1514] Processing DeleteTablet for tablet e4eaa94c570047bc921189a3b81e1525 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:52:07 UTC) from {username='kudu', principal='kudu/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:52294
I20260806 02:52:07.826468 29782 tablet_service.cc:1514] Processing DeleteTablet for tablet e4eaa94c570047bc921189a3b81e1525 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:52:07 UTC) from {username='kudu', principal='kudu/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:60664
I20260806 02:52:07.827047 29576 tablet_service.cc:1514] Processing DeleteTablet for tablet e4eaa94c570047bc921189a3b81e1525 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:52:07 UTC) from {username='kudu', principal='kudu/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:41152
I20260806 02:52:07.827128 30019 tablet_replica.cc:330] T e4eaa94c570047bc921189a3b81e1525 P b5272675298b46c4b9bea5414bd422b4: stopping tablet replica
I20260806 02:52:07.827366 30019 raft_consensus.cc:2229] T e4eaa94c570047bc921189a3b81e1525 P b5272675298b46c4b9bea5414bd422b4 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:52:07.827838 30019 raft_consensus.cc:2258] T e4eaa94c570047bc921189a3b81e1525 P b5272675298b46c4b9bea5414bd422b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:52:07.829536 30019 ts_tablet_manager.cc:1905] T e4eaa94c570047bc921189a3b81e1525 P b5272675298b46c4b9bea5414bd422b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:52:07.838076 30022 tablet_replica.cc:330] T e4eaa94c570047bc921189a3b81e1525 P 9d1c9b0b0124405398afc3033537cf76: stopping tablet replica
I20260806 02:52:07.838478 30022 raft_consensus.cc:2229] T e4eaa94c570047bc921189a3b81e1525 P 9d1c9b0b0124405398afc3033537cf76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:52:07.838747 30019 ts_tablet_manager.cc:1918] T e4eaa94c570047bc921189a3b81e1525 P b5272675298b46c4b9bea5414bd422b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 02:52:07.838794 30022 raft_consensus.cc:2258] T e4eaa94c570047bc921189a3b81e1525 P 9d1c9b0b0124405398afc3033537cf76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:52:07.839005 30019 log.cc:1196] T e4eaa94c570047bc921189a3b81e1525 P b5272675298b46c4b9bea5414bd422b4: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/ts-0/wal/wals/e4eaa94c570047bc921189a3b81e1525
I20260806 02:52:07.839344 30020 tablet_replica.cc:330] T e4eaa94c570047bc921189a3b81e1525 P d3114496d23749e48a2889e0a3ceef40: stopping tablet replica
I20260806 02:52:07.839617 30019 ts_tablet_manager.cc:1939] T e4eaa94c570047bc921189a3b81e1525 P b5272675298b46c4b9bea5414bd422b4: Deleting consensus metadata
I20260806 02:52:07.839593 30020 raft_consensus.cc:2229] T e4eaa94c570047bc921189a3b81e1525 P d3114496d23749e48a2889e0a3ceef40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:52:07.839845 30020 raft_consensus.cc:2258] T e4eaa94c570047bc921189a3b81e1525 P d3114496d23749e48a2889e0a3ceef40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:52:07.839998 30022 ts_tablet_manager.cc:1905] T e4eaa94c570047bc921189a3b81e1525 P 9d1c9b0b0124405398afc3033537cf76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:52:07.840830 30020 ts_tablet_manager.cc:1905] T e4eaa94c570047bc921189a3b81e1525 P d3114496d23749e48a2889e0a3ceef40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:52:07.841037 29151 catalog_manager.cc:4841] TS b5272675298b46c4b9bea5414bd422b4 (127.25.211.129:33379): tablet e4eaa94c570047bc921189a3b81e1525 (table test-table [id=8cfe894039f3460381c6e51341be8de5]) successfully deleted
I20260806 02:52:07.845458 30022 ts_tablet_manager.cc:1918] T e4eaa94c570047bc921189a3b81e1525 P 9d1c9b0b0124405398afc3033537cf76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 02:52:07.845626 30022 log.cc:1196] T e4eaa94c570047bc921189a3b81e1525 P 9d1c9b0b0124405398afc3033537cf76: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/ts-1/wal/wals/e4eaa94c570047bc921189a3b81e1525
I20260806 02:52:07.846040 30022 ts_tablet_manager.cc:1939] T e4eaa94c570047bc921189a3b81e1525 P 9d1c9b0b0124405398afc3033537cf76: Deleting consensus metadata
I20260806 02:52:07.847007 30020 ts_tablet_manager.cc:1918] T e4eaa94c570047bc921189a3b81e1525 P d3114496d23749e48a2889e0a3ceef40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 02:52:07.847208 29151 catalog_manager.cc:4841] TS 9d1c9b0b0124405398afc3033537cf76 (127.25.211.130:46561): tablet e4eaa94c570047bc921189a3b81e1525 (table test-table [id=8cfe894039f3460381c6e51341be8de5]) successfully deleted
I20260806 02:52:07.847295 30020 log.cc:1196] T e4eaa94c570047bc921189a3b81e1525 P d3114496d23749e48a2889e0a3ceef40: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/ts-2/wal/wals/e4eaa94c570047bc921189a3b81e1525
I20260806 02:52:07.847790 30020 ts_tablet_manager.cc:1939] T e4eaa94c570047bc921189a3b81e1525 P d3114496d23749e48a2889e0a3ceef40: Deleting consensus metadata
I20260806 02:52:07.849053 29150 catalog_manager.cc:4841] TS d3114496d23749e48a2889e0a3ceef40 (127.25.211.131:45757): tablet e4eaa94c570047bc921189a3b81e1525 (table test-table [id=8cfe894039f3460381c6e51341be8de5]) successfully deleted
Aug 06 02:52:07 dist-test-slave-zj8q krb5kdc[29106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984727, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 02:52:07 dist-test-slave-zj8q krb5kdc[29106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984727, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:52:07.893587 30032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:07.877385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48584 (local address 127.25.211.190:38929)
0806 02:52:07.878391 (+ 1006us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:07.878396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:07.878443 (+ 47us) server_negotiation.cc:407] Connection header received
0806 02:52:07.878502 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:07.878525 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:07.878610 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:07.878699 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:07.880203 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:07.881081 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:07.882332 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:07.882758 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:07.882852 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:07.887360 (+ 4508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:07.887367 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:07.887370 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:07.887419 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:07.890739 (+ 3320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:07.891702 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:07.891727 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:07.891729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:07.891864 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:07.892245 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:07.892249 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:07.892251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:07.892539 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:07.892754 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:07.893011 (+ 257us) server_negotiation.cc:299] Negotiation successful
0806 02:52:07.893237 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":892,"thread_start_us":750,"threads_started":1}
I20260806 02:52:07.897202 29170 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48584:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:52:07.897835 29170 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:52:07.904881 29380 tablet_service.cc:1467] Processing CreateTablet for tablet 5e77058aece64558956bb36f8f17f4b1 (DEFAULT_TABLE table=test-table [id=681f443124524eae949dbfca145edb81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:52:07.904881 29576 tablet_service.cc:1467] Processing CreateTablet for tablet 5e77058aece64558956bb36f8f17f4b1 (DEFAULT_TABLE table=test-table [id=681f443124524eae949dbfca145edb81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:52:07.905115 29782 tablet_service.cc:1467] Processing CreateTablet for tablet 5e77058aece64558956bb36f8f17f4b1 (DEFAULT_TABLE table=test-table [id=681f443124524eae949dbfca145edb81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:52:07.905382 29576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e77058aece64558956bb36f8f17f4b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:07.905407 29380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e77058aece64558956bb36f8f17f4b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:07.905571 29782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e77058aece64558956bb36f8f17f4b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:07.911875 30033 tablet_bootstrap.cc:492] T 5e77058aece64558956bb36f8f17f4b1 P d3114496d23749e48a2889e0a3ceef40: Bootstrap starting.
I20260806 02:52:07.912607 30035 tablet_bootstrap.cc:492] T 5e77058aece64558956bb36f8f17f4b1 P b5272675298b46c4b9bea5414bd422b4: Bootstrap starting.
I20260806 02:52:07.913403 30037 tablet_bootstrap.cc:492] T 5e77058aece64558956bb36f8f17f4b1 P 9d1c9b0b0124405398afc3033537cf76: Bootstrap starting.
I20260806 02:52:07.913859 30033 tablet_bootstrap.cc:654] T 5e77058aece64558956bb36f8f17f4b1 P d3114496d23749e48a2889e0a3ceef40: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:07.914772 30035 tablet_bootstrap.cc:654] T 5e77058aece64558956bb36f8f17f4b1 P b5272675298b46c4b9bea5414bd422b4: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:07.915812 30037 tablet_bootstrap.cc:654] T 5e77058aece64558956bb36f8f17f4b1 P 9d1c9b0b0124405398afc3033537cf76: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:07.916630 30033 tablet_bootstrap.cc:492] T 5e77058aece64558956bb36f8f17f4b1 P d3114496d23749e48a2889e0a3ceef40: No bootstrap required, opened a new log
I20260806 02:52:07.916779 30033 ts_tablet_manager.cc:1397] T 5e77058aece64558956bb36f8f17f4b1 P d3114496d23749e48a2889e0a3ceef40: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 02:52:07.917433 30033 raft_consensus.cc:348] T 5e77058aece64558956bb36f8f17f4b1 P d3114496d23749e48a2889e0a3ceef40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5272675298b46c4b9bea5414bd422b4" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 33379 } } peers { permanent_uuid: "9d1c9b0b0124405398afc3033537cf76" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46561 } } peers { permanent_uuid: "d3114496d23749e48a2889e0a3ceef40" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45757 } }
I20260806 02:52:07.917647 30033 raft_consensus.cc:374] T 5e77058aece64558956bb36f8f17f4b1 P d3114496d23749e48a2889e0a3ceef40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:07.917739 30033 raft_consensus.cc:729] T 5e77058aece64558956bb36f8f17f4b1 P d3114496d23749e48a2889e0a3ceef40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3114496d23749e48a2889e0a3ceef40, State: Initialized, Role: FOLLOWER
I20260806 02:52:07.917980 30033 consensus_queue.cc:260] T 5e77058aece64558956bb36f8f17f4b1 P d3114496d23749e48a2889e0a3ceef40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5272675298b46c4b9bea5414bd422b4" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 33379 } } peers { permanent_uuid: "9d1c9b0b0124405398afc3033537cf76" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46561 } } peers { permanent_uuid: "d3114496d23749e48a2889e0a3ceef40" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45757 } }
I20260806 02:52:07.918815 30037 tablet_bootstrap.cc:492] T 5e77058aece64558956bb36f8f17f4b1 P 9d1c9b0b0124405398afc3033537cf76: No bootstrap required, opened a new log
I20260806 02:52:07.918846 30033 ts_tablet_manager.cc:1428] T 5e77058aece64558956bb36f8f17f4b1 P d3114496d23749e48a2889e0a3ceef40: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 02:52:07.919063 30037 ts_tablet_manager.cc:1397] T 5e77058aece64558956bb36f8f17f4b1 P 9d1c9b0b0124405398afc3033537cf76: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260806 02:52:07.920260 30035 tablet_bootstrap.cc:492] T 5e77058aece64558956bb36f8f17f4b1 P b5272675298b46c4b9bea5414bd422b4: No bootstrap required, opened a new log
I20260806 02:52:07.920176 30037 raft_consensus.cc:348] T 5e77058aece64558956bb36f8f17f4b1 P 9d1c9b0b0124405398afc3033537cf76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5272675298b46c4b9bea5414bd422b4" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 33379 } } peers { permanent_uuid: "9d1c9b0b0124405398afc3033537cf76" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46561 } } peers { permanent_uuid: "d3114496d23749e48a2889e0a3ceef40" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45757 } }
I20260806 02:52:07.920483 30035 ts_tablet_manager.cc:1397] T 5e77058aece64558956bb36f8f17f4b1 P b5272675298b46c4b9bea5414bd422b4: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260806 02:52:07.920501 30037 raft_consensus.cc:374] T 5e77058aece64558956bb36f8f17f4b1 P 9d1c9b0b0124405398afc3033537cf76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:07.920936 30037 raft_consensus.cc:729] T 5e77058aece64558956bb36f8f17f4b1 P 9d1c9b0b0124405398afc3033537cf76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d1c9b0b0124405398afc3033537cf76, State: Initialized, Role: FOLLOWER
I20260806 02:52:07.921289 30037 consensus_queue.cc:260] T 5e77058aece64558956bb36f8f17f4b1 P 9d1c9b0b0124405398afc3033537cf76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5272675298b46c4b9bea5414bd422b4" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 33379 } } peers { permanent_uuid: "9d1c9b0b0124405398afc3033537cf76" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46561 } } peers { permanent_uuid: "d3114496d23749e48a2889e0a3ceef40" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45757 } }
I20260806 02:52:07.921309 30035 raft_consensus.cc:348] T 5e77058aece64558956bb36f8f17f4b1 P b5272675298b46c4b9bea5414bd422b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5272675298b46c4b9bea5414bd422b4" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 33379 } } peers { permanent_uuid: "9d1c9b0b0124405398afc3033537cf76" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46561 } } peers { permanent_uuid: "d3114496d23749e48a2889e0a3ceef40" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45757 } }
I20260806 02:52:07.921605 30035 raft_consensus.cc:374] T 5e77058aece64558956bb36f8f17f4b1 P b5272675298b46c4b9bea5414bd422b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:07.921792 30035 raft_consensus.cc:729] T 5e77058aece64558956bb36f8f17f4b1 P b5272675298b46c4b9bea5414bd422b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5272675298b46c4b9bea5414bd422b4, State: Initialized, Role: FOLLOWER
I20260806 02:52:07.922335 30037 ts_tablet_manager.cc:1428] T 5e77058aece64558956bb36f8f17f4b1 P 9d1c9b0b0124405398afc3033537cf76: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 02:52:07.922747 30035 consensus_queue.cc:260] T 5e77058aece64558956bb36f8f17f4b1 P b5272675298b46c4b9bea5414bd422b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5272675298b46c4b9bea5414bd422b4" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 33379 } } peers { permanent_uuid: "9d1c9b0b0124405398afc3033537cf76" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46561 } } peers { permanent_uuid: "d3114496d23749e48a2889e0a3ceef40" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45757 } }
I20260806 02:52:07.923364 30035 ts_tablet_manager.cc:1428] T 5e77058aece64558956bb36f8f17f4b1 P b5272675298b46c4b9bea5414bd422b4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 02:52:08.158052 30041 raft_consensus.cc:482] T 5e77058aece64558956bb36f8f17f4b1 P d3114496d23749e48a2889e0a3ceef40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:52:08.158353 30041 raft_consensus.cc:504] T 5e77058aece64558956bb36f8f17f4b1 P d3114496d23749e48a2889e0a3ceef40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5272675298b46c4b9bea5414bd422b4" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 33379 } } peers { permanent_uuid: "9d1c9b0b0124405398afc3033537cf76" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46561 } } peers { permanent_uuid: "d3114496d23749e48a2889e0a3ceef40" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45757 } }
I20260806 02:52:08.160071 30041 leader_election.cc:288] T 5e77058aece64558956bb36f8f17f4b1 P d3114496d23749e48a2889e0a3ceef40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5272675298b46c4b9bea5414bd422b4 (127.25.211.129:33379), 9d1c9b0b0124405398afc3033537cf76 (127.25.211.130:46561)
I20260806 02:52:08.170086 30048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:08.161429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:50473 (local address 127.25.211.130:46561)
0806 02:52:08.162220 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:08.162233 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:08.162252 (+ 19us) server_negotiation.cc:407] Connection header received
0806 02:52:08.162310 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:08.162313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:08.162381 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:08.162468 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:52:08.163399 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:08.164646 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:08.167672 (+ 3026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:08.169012 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:08.169263 (+ 251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:08.169347 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:08.169586 (+ 239us) server_negotiation.cc:299] Negotiation successful
0806 02:52:08.169736 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":455,"threads_started":1}
I20260806 02:52:08.170212 30047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:08.161267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.130:46561 (local address 127.25.211.131:50473)
0806 02:52:08.162042 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:08.162058 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:08.162194 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:08.162522 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:08.162526 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:08.162550 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:52:08.163203 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:08.163214 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:08.164806 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:08.164813 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:08.167483 (+ 2670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:08.167498 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:08.169261 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:08.169267 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:08.169399 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:08.169428 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 02:52:08.169549 (+ 121us) client_negotiation.cc:240] Negotiation successful
0806 02:52:08.169707 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"mutex_wait_us":32,"spinlock_wait_cycles":1408,"thread_start_us":292,"threads_started":1}
I20260806 02:52:08.170604 30045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:08.160422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:33379 (local address 127.25.211.131:45977)
0806 02:52:08.161138 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:08.161161 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:08.161325 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:08.162066 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:08.162070 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:08.162100 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:52:08.162749 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:08.162765 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:08.164887 (+ 2122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:08.164892 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:08.167476 (+ 2584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:08.167499 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:08.169635 (+ 2136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:08.169642 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:08.169815 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:08.169851 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 02:52:08.169940 (+ 89us) client_negotiation.cc:240] Negotiation successful
0806 02:52:08.170095 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"mutex_wait_us":31,"spinlock_wait_cycles":2432,"thread_start_us":381,"threads_started":1}
I20260806 02:52:08.170615 30046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:08.160544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:45977 (local address 127.25.211.129:33379)
0806 02:52:08.161577 (+ 1033us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:08.161585 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:08.161613 (+ 28us) server_negotiation.cc:407] Connection header received
0806 02:52:08.161694 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:08.161699 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:08.161847 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:08.161960 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:52:08.162979 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:08.164619 (+ 1640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:08.167684 (+ 3065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:08.169388 (+ 1704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:08.169595 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:08.169673 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:08.170134 (+ 461us) server_negotiation.cc:299] Negotiation successful
0806 02:52:08.170313 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":771,"thread_start_us":518,"threads_started":1}
I20260806 02:52:08.171155 29612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e77058aece64558956bb36f8f17f4b1" candidate_uuid: "d3114496d23749e48a2889e0a3ceef40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d1c9b0b0124405398afc3033537cf76" is_pre_election: true
I20260806 02:52:08.171435 29612 raft_consensus.cc:2454] T 5e77058aece64558956bb36f8f17f4b1 P 9d1c9b0b0124405398afc3033537cf76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3114496d23749e48a2889e0a3ceef40 in term 0.
I20260806 02:52:08.171478 29410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e77058aece64558956bb36f8f17f4b1" candidate_uuid: "d3114496d23749e48a2889e0a3ceef40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5272675298b46c4b9bea5414bd422b4" is_pre_election: true
I20260806 02:52:08.171978 29410 raft_consensus.cc:2454] T 5e77058aece64558956bb36f8f17f4b1 P b5272675298b46c4b9bea5414bd422b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3114496d23749e48a2889e0a3ceef40 in term 0.
I20260806 02:52:08.171964 29709 leader_election.cc:302] T 5e77058aece64558956bb36f8f17f4b1 P d3114496d23749e48a2889e0a3ceef40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d1c9b0b0124405398afc3033537cf76, d3114496d23749e48a2889e0a3ceef40; no voters:
I20260806 02:52:08.172406 30041 raft_consensus.cc:2781] T 5e77058aece64558956bb36f8f17f4b1 P d3114496d23749e48a2889e0a3ceef40 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:52:08.172515 30041 raft_consensus.cc:482] T 5e77058aece64558956bb36f8f17f4b1 P d3114496d23749e48a2889e0a3ceef40 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:52:08.172614 30041 raft_consensus.cc:3037] T 5e77058aece64558956bb36f8f17f4b1 P d3114496d23749e48a2889e0a3ceef40 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:08.175115 30041 raft_consensus.cc:504] T 5e77058aece64558956bb36f8f17f4b1 P d3114496d23749e48a2889e0a3ceef40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5272675298b46c4b9bea5414bd422b4" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 33379 } } peers { permanent_uuid: "9d1c9b0b0124405398afc3033537cf76" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46561 } } peers { permanent_uuid: "d3114496d23749e48a2889e0a3ceef40" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45757 } }
I20260806 02:52:08.175769 30041 leader_election.cc:288] T 5e77058aece64558956bb36f8f17f4b1 P d3114496d23749e48a2889e0a3ceef40 [CANDIDATE]: Term 1 election: Requested vote from peers b5272675298b46c4b9bea5414bd422b4 (127.25.211.129:33379), 9d1c9b0b0124405398afc3033537cf76 (127.25.211.130:46561)
I20260806 02:52:08.176249 29410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e77058aece64558956bb36f8f17f4b1" candidate_uuid: "d3114496d23749e48a2889e0a3ceef40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5272675298b46c4b9bea5414bd422b4"
I20260806 02:52:08.176386 29612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e77058aece64558956bb36f8f17f4b1" candidate_uuid: "d3114496d23749e48a2889e0a3ceef40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d1c9b0b0124405398afc3033537cf76"
I20260806 02:52:08.176450 29410 raft_consensus.cc:3037] T 5e77058aece64558956bb36f8f17f4b1 P b5272675298b46c4b9bea5414bd422b4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:08.176601 29612 raft_consensus.cc:3037] T 5e77058aece64558956bb36f8f17f4b1 P 9d1c9b0b0124405398afc3033537cf76 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:08.178840 29612 raft_consensus.cc:2454] T 5e77058aece64558956bb36f8f17f4b1 P 9d1c9b0b0124405398afc3033537cf76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3114496d23749e48a2889e0a3ceef40 in term 1.
I20260806 02:52:08.178956 29410 raft_consensus.cc:2454] T 5e77058aece64558956bb36f8f17f4b1 P b5272675298b46c4b9bea5414bd422b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3114496d23749e48a2889e0a3ceef40 in term 1.
I20260806 02:52:08.179328 29709 leader_election.cc:302] T 5e77058aece64558956bb36f8f17f4b1 P d3114496d23749e48a2889e0a3ceef40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d1c9b0b0124405398afc3033537cf76, d3114496d23749e48a2889e0a3ceef40; no voters:
I20260806 02:52:08.179611 30041 raft_consensus.cc:2781] T 5e77058aece64558956bb36f8f17f4b1 P d3114496d23749e48a2889e0a3ceef40 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:52:08.180130 30041 raft_consensus.cc:686] T 5e77058aece64558956bb36f8f17f4b1 P d3114496d23749e48a2889e0a3ceef40 [term 1 LEADER]: Becoming Leader. State: Replica: d3114496d23749e48a2889e0a3ceef40, State: Running, Role: LEADER
I20260806 02:52:08.180524 30041 consensus_queue.cc:237] T 5e77058aece64558956bb36f8f17f4b1 P d3114496d23749e48a2889e0a3ceef40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5272675298b46c4b9bea5414bd422b4" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 33379 } } peers { permanent_uuid: "9d1c9b0b0124405398afc3033537cf76" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46561 } } peers { permanent_uuid: "d3114496d23749e48a2889e0a3ceef40" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45757 } }
I20260806 02:52:08.184092 29171 catalog_manager.cc:5495] T 5e77058aece64558956bb36f8f17f4b1 P d3114496d23749e48a2889e0a3ceef40 reported cstate change: term changed from 0 to 1, leader changed from <none> to d3114496d23749e48a2889e0a3ceef40 (127.25.211.131). New cstate: current_term: 1 leader_uuid: "d3114496d23749e48a2889e0a3ceef40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5272675298b46c4b9bea5414bd422b4" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 33379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d1c9b0b0124405398afc3033537cf76" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3114496d23749e48a2889e0a3ceef40" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45757 } health_report { overall_health: HEALTHY } } }
I20260806 02:52:08.225977 30055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:08.218747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60680 (local address 127.25.211.131:45757)
0806 02:52:08.219353 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:08.219357 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:08.219376 (+ 19us) server_negotiation.cc:407] Connection header received
0806 02:52:08.219463 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:08.219466 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:08.219560 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:08.219712 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:52:08.220221 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:08.221612 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:08.223191 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:08.223888 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:08.224004 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:08.224447 (+ 443us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:52:08.225271 (+ 824us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:52:08.225411 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:08.225480 (+ 69us) server_negotiation.cc:299] Negotiation successful
0806 02:52:08.225582 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":355,"threads_started":1}
I20260806 02:52:08.244415 30046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:08.238200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:56337 (local address 127.25.211.129:33379)
0806 02:52:08.238398 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:08.238403 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:08.239118 (+ 715us) server_negotiation.cc:407] Connection header received
0806 02:52:08.239312 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:08.239316 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:08.239385 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:08.239461 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:52:08.240081 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:08.240926 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:08.242696 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:08.243118 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:08.243241 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:08.243734 (+ 493us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:52:08.243840 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:52:08.243914 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:08.244046 (+ 132us) server_negotiation.cc:299] Negotiation successful
0806 02:52:08.244130 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 02:52:08.244616 30061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:08.238212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:33379 (local address 127.25.211.131:56337)
0806 02:52:08.239003 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:08.239020 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:08.239175 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:08.239497 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:08.239501 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:08.239512 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:52:08.239923 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:08.239935 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:08.241064 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:08.241069 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:08.242557 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:08.242572 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:08.243352 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:08.243358 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:08.243496 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:08.243566 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:52:08.243934 (+ 368us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:52:08.243946 (+ 12us) client_negotiation.cc:769] Sending connection context
0806 02:52:08.244013 (+ 67us) client_negotiation.cc:240] Negotiation successful
0806 02:52:08.244213 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":432,"threads_started":1}
I20260806 02:52:08.254130 29612 raft_consensus.cc:1261] T 5e77058aece64558956bb36f8f17f4b1 P 9d1c9b0b0124405398afc3033537cf76 [term 1 FOLLOWER]: Refusing update from remote peer d3114496d23749e48a2889e0a3ceef40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:52:08.254593 29410 raft_consensus.cc:1261] T 5e77058aece64558956bb36f8f17f4b1 P b5272675298b46c4b9bea5414bd422b4 [term 1 FOLLOWER]: Refusing update from remote peer d3114496d23749e48a2889e0a3ceef40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:52:08.255059 30052 consensus_queue.cc:1035] T 5e77058aece64558956bb36f8f17f4b1 P d3114496d23749e48a2889e0a3ceef40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d1c9b0b0124405398afc3033537cf76" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:52:08.255661 30041 consensus_queue.cc:1035] T 5e77058aece64558956bb36f8f17f4b1 P d3114496d23749e48a2889e0a3ceef40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5272675298b46c4b9bea5414bd422b4" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 33379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:52:08.262705 30002 mvcc.cc:205] Tried to move back new op lower bound from 7315393446922014720 to 7315393446630961152. Current Snapshot: MvccSnapshot[applied={T|T < 7315393446922014720}]
I20260806 02:52:08.285817 30055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:08.279731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:44903 (local address 127.25.211.131:45757)
0806 02:52:08.279884 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:08.279889 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:08.280578 (+ 689us) server_negotiation.cc:407] Connection header received
0806 02:52:08.280760 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:08.280765 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:08.280831 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:08.280904 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:52:08.281409 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:08.282228 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:08.284235 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:08.284736 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:08.284835 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:08.285225 (+ 390us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:52:08.285330 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:52:08.285394 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:08.285528 (+ 134us) server_negotiation.cc:299] Negotiation successful
0806 02:52:08.285598 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 02:52:08.285913 30066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:08.279768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.131:45757 (local address 127.25.211.129:44903)
0806 02:52:08.280472 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:08.280489 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:08.280623 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:08.280932 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:08.280937 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:08.280948 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:52:08.281251 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:08.281262 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:08.282377 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:08.282381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:08.284052 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:08.284068 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:08.284902 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:08.284907 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:08.285036 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:08.285074 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:52:08.285416 (+ 342us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:52:08.285426 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 02:52:08.285490 (+ 64us) client_negotiation.cc:240] Negotiation successful
0806 02:52:08.285619 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":415,"threads_started":1}
I20260806 02:52:08.318225 29171 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:52:08.318493 29171 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:52:08.322012 29171 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 50d796ee82c3472496eacc2910ede519: Sending DeleteTablet for 3 replicas of tablet 5e77058aece64558956bb36f8f17f4b1
I20260806 02:52:08.323089 29576 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e77058aece64558956bb36f8f17f4b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:52:08 UTC) from {username='kudu', principal='kudu/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:41152
I20260806 02:52:08.323164 29782 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e77058aece64558956bb36f8f17f4b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:52:08 UTC) from {username='kudu', principal='kudu/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:60664
I20260806 02:52:08.323434 30022 tablet_replica.cc:330] T 5e77058aece64558956bb36f8f17f4b1 P 9d1c9b0b0124405398afc3033537cf76: stopping tablet replica
I20260806 02:52:08.323462 30020 tablet_replica.cc:330] T 5e77058aece64558956bb36f8f17f4b1 P d3114496d23749e48a2889e0a3ceef40: stopping tablet replica
I20260806 02:52:08.323629 30020 raft_consensus.cc:2229] T 5e77058aece64558956bb36f8f17f4b1 P d3114496d23749e48a2889e0a3ceef40 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:52:08.323693 30022 raft_consensus.cc:2229] T 5e77058aece64558956bb36f8f17f4b1 P 9d1c9b0b0124405398afc3033537cf76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:52:08.323930 30022 raft_consensus.cc:2258] T 5e77058aece64558956bb36f8f17f4b1 P 9d1c9b0b0124405398afc3033537cf76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:52:08.324039 30020 raft_consensus.cc:2258] T 5e77058aece64558956bb36f8f17f4b1 P d3114496d23749e48a2889e0a3ceef40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:52:08.324559 29380 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e77058aece64558956bb36f8f17f4b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:52:08 UTC) from {username='kudu', principal='kudu/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:52294
I20260806 02:52:08.324887 30019 tablet_replica.cc:330] T 5e77058aece64558956bb36f8f17f4b1 P b5272675298b46c4b9bea5414bd422b4: stopping tablet replica
I20260806 02:52:08.325114 30019 raft_consensus.cc:2229] T 5e77058aece64558956bb36f8f17f4b1 P b5272675298b46c4b9bea5414bd422b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:52:08.325251 30020 ts_tablet_manager.cc:1905] T 5e77058aece64558956bb36f8f17f4b1 P d3114496d23749e48a2889e0a3ceef40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:52:08.325340 30019 raft_consensus.cc:2258] T 5e77058aece64558956bb36f8f17f4b1 P b5272675298b46c4b9bea5414bd422b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:52:08.326560 30019 ts_tablet_manager.cc:1905] T 5e77058aece64558956bb36f8f17f4b1 P b5272675298b46c4b9bea5414bd422b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:52:08.330428 29171 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48584:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260806 02:52:08.330649 30022 ts_tablet_manager.cc:1905] T 5e77058aece64558956bb36f8f17f4b1 P 9d1c9b0b0124405398afc3033537cf76: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260806 02:52:08.331534 29171 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:52:08.332122 30020 ts_tablet_manager.cc:1918] T 5e77058aece64558956bb36f8f17f4b1 P d3114496d23749e48a2889e0a3ceef40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:52:08.332396 30020 log.cc:1196] T 5e77058aece64558956bb36f8f17f4b1 P d3114496d23749e48a2889e0a3ceef40: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/ts-2/wal/wals/5e77058aece64558956bb36f8f17f4b1
I20260806 02:52:08.333078 30020 ts_tablet_manager.cc:1939] T 5e77058aece64558956bb36f8f17f4b1 P d3114496d23749e48a2889e0a3ceef40: Deleting consensus metadata
I20260806 02:52:08.334395 29150 catalog_manager.cc:4841] TS d3114496d23749e48a2889e0a3ceef40 (127.25.211.131:45757): tablet 5e77058aece64558956bb36f8f17f4b1 (table test-table [id=681f443124524eae949dbfca145edb81]) successfully deleted
I20260806 02:52:08.338485 30019 ts_tablet_manager.cc:1918] T 5e77058aece64558956bb36f8f17f4b1 P b5272675298b46c4b9bea5414bd422b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:52:08.338830 30019 log.cc:1196] T 5e77058aece64558956bb36f8f17f4b1 P b5272675298b46c4b9bea5414bd422b4: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/ts-0/wal/wals/5e77058aece64558956bb36f8f17f4b1
I20260806 02:52:08.339258 30022 ts_tablet_manager.cc:1918] T 5e77058aece64558956bb36f8f17f4b1 P 9d1c9b0b0124405398afc3033537cf76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:52:08.339437 30022 log.cc:1196] T 5e77058aece64558956bb36f8f17f4b1 P 9d1c9b0b0124405398afc3033537cf76: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/ts-1/wal/wals/5e77058aece64558956bb36f8f17f4b1
I20260806 02:52:08.339452 30019 ts_tablet_manager.cc:1939] T 5e77058aece64558956bb36f8f17f4b1 P b5272675298b46c4b9bea5414bd422b4: Deleting consensus metadata
I20260806 02:52:08.339974 30022 ts_tablet_manager.cc:1939] T 5e77058aece64558956bb36f8f17f4b1 P 9d1c9b0b0124405398afc3033537cf76: Deleting consensus metadata
I20260806 02:52:08.340919 29151 catalog_manager.cc:4841] TS b5272675298b46c4b9bea5414bd422b4 (127.25.211.129:33379): tablet 5e77058aece64558956bb36f8f17f4b1 (table test-table [id=681f443124524eae949dbfca145edb81]) successfully deleted
I20260806 02:52:08.341552 29151 catalog_manager.cc:4841] TS 9d1c9b0b0124405398afc3033537cf76 (127.25.211.130:46561): tablet 5e77058aece64558956bb36f8f17f4b1 (table test-table [id=681f443124524eae949dbfca145edb81]) successfully deleted
I20260806 02:52:08.342453 29576 tablet_service.cc:1467] Processing CreateTablet for tablet 62d24ea6f13d4492974018c1b869ca0f (DEFAULT_TABLE table=test-table [id=22e39ee0aea04808a1717a36e6314fab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:52:08.342594 29782 tablet_service.cc:1467] Processing CreateTablet for tablet 62d24ea6f13d4492974018c1b869ca0f (DEFAULT_TABLE table=test-table [id=22e39ee0aea04808a1717a36e6314fab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:52:08.342900 29576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62d24ea6f13d4492974018c1b869ca0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:08.343179 29782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62d24ea6f13d4492974018c1b869ca0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:08.344190 29380 tablet_service.cc:1467] Processing CreateTablet for tablet 62d24ea6f13d4492974018c1b869ca0f (DEFAULT_TABLE table=test-table [id=22e39ee0aea04808a1717a36e6314fab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:52:08.344691 29380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62d24ea6f13d4492974018c1b869ca0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:08.349550 30037 tablet_bootstrap.cc:492] T 62d24ea6f13d4492974018c1b869ca0f P 9d1c9b0b0124405398afc3033537cf76: Bootstrap starting.
I20260806 02:52:08.351897 30037 tablet_bootstrap.cc:654] T 62d24ea6f13d4492974018c1b869ca0f P 9d1c9b0b0124405398afc3033537cf76: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:08.352047 30033 tablet_bootstrap.cc:492] T 62d24ea6f13d4492974018c1b869ca0f P d3114496d23749e48a2889e0a3ceef40: Bootstrap starting.
I20260806 02:52:08.353040 30035 tablet_bootstrap.cc:492] T 62d24ea6f13d4492974018c1b869ca0f P b5272675298b46c4b9bea5414bd422b4: Bootstrap starting.
I20260806 02:52:08.354110 30033 tablet_bootstrap.cc:654] T 62d24ea6f13d4492974018c1b869ca0f P d3114496d23749e48a2889e0a3ceef40: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:08.355464 30035 tablet_bootstrap.cc:654] T 62d24ea6f13d4492974018c1b869ca0f P b5272675298b46c4b9bea5414bd422b4: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:08.357247 30037 tablet_bootstrap.cc:492] T 62d24ea6f13d4492974018c1b869ca0f P 9d1c9b0b0124405398afc3033537cf76: No bootstrap required, opened a new log
I20260806 02:52:08.357458 30037 ts_tablet_manager.cc:1397] T 62d24ea6f13d4492974018c1b869ca0f P 9d1c9b0b0124405398afc3033537cf76: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260806 02:52:08.358434 30037 raft_consensus.cc:348] T 62d24ea6f13d4492974018c1b869ca0f P 9d1c9b0b0124405398afc3033537cf76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5272675298b46c4b9bea5414bd422b4" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 33379 } } peers { permanent_uuid: "9d1c9b0b0124405398afc3033537cf76" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46561 } } peers { permanent_uuid: "d3114496d23749e48a2889e0a3ceef40" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45757 } }
I20260806 02:52:08.358762 30037 raft_consensus.cc:374] T 62d24ea6f13d4492974018c1b869ca0f P 9d1c9b0b0124405398afc3033537cf76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:08.358836 30037 raft_consensus.cc:729] T 62d24ea6f13d4492974018c1b869ca0f P 9d1c9b0b0124405398afc3033537cf76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d1c9b0b0124405398afc3033537cf76, State: Initialized, Role: FOLLOWER
I20260806 02:52:08.359092 30037 consensus_queue.cc:260] T 62d24ea6f13d4492974018c1b869ca0f P 9d1c9b0b0124405398afc3033537cf76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5272675298b46c4b9bea5414bd422b4" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 33379 } } peers { permanent_uuid: "9d1c9b0b0124405398afc3033537cf76" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46561 } } peers { permanent_uuid: "d3114496d23749e48a2889e0a3ceef40" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45757 } }
I20260806 02:52:08.359838 30037 ts_tablet_manager.cc:1428] T 62d24ea6f13d4492974018c1b869ca0f P 9d1c9b0b0124405398afc3033537cf76: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 02:52:08.359872 30033 tablet_bootstrap.cc:492] T 62d24ea6f13d4492974018c1b869ca0f P d3114496d23749e48a2889e0a3ceef40: No bootstrap required, opened a new log
I20260806 02:52:08.360049 30033 ts_tablet_manager.cc:1397] T 62d24ea6f13d4492974018c1b869ca0f P d3114496d23749e48a2889e0a3ceef40: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260806 02:52:08.360908 30033 raft_consensus.cc:348] T 62d24ea6f13d4492974018c1b869ca0f P d3114496d23749e48a2889e0a3ceef40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5272675298b46c4b9bea5414bd422b4" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 33379 } } peers { permanent_uuid: "9d1c9b0b0124405398afc3033537cf76" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46561 } } peers { permanent_uuid: "d3114496d23749e48a2889e0a3ceef40" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45757 } }
I20260806 02:52:08.361212 30033 raft_consensus.cc:374] T 62d24ea6f13d4492974018c1b869ca0f P d3114496d23749e48a2889e0a3ceef40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:08.361402 30033 raft_consensus.cc:729] T 62d24ea6f13d4492974018c1b869ca0f P d3114496d23749e48a2889e0a3ceef40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3114496d23749e48a2889e0a3ceef40, State: Initialized, Role: FOLLOWER
I20260806 02:52:08.362102 30033 consensus_queue.cc:260] T 62d24ea6f13d4492974018c1b869ca0f P d3114496d23749e48a2889e0a3ceef40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5272675298b46c4b9bea5414bd422b4" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 33379 } } peers { permanent_uuid: "9d1c9b0b0124405398afc3033537cf76" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46561 } } peers { permanent_uuid: "d3114496d23749e48a2889e0a3ceef40" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45757 } }
I20260806 02:52:08.362882 30033 ts_tablet_manager.cc:1428] T 62d24ea6f13d4492974018c1b869ca0f P d3114496d23749e48a2889e0a3ceef40: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260806 02:52:08.369437 30035 tablet_bootstrap.cc:492] T 62d24ea6f13d4492974018c1b869ca0f P b5272675298b46c4b9bea5414bd422b4: No bootstrap required, opened a new log
I20260806 02:52:08.369642 30035 ts_tablet_manager.cc:1397] T 62d24ea6f13d4492974018c1b869ca0f P b5272675298b46c4b9bea5414bd422b4: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.009s
I20260806 02:52:08.370489 30035 raft_consensus.cc:348] T 62d24ea6f13d4492974018c1b869ca0f P b5272675298b46c4b9bea5414bd422b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5272675298b46c4b9bea5414bd422b4" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 33379 } } peers { permanent_uuid: "9d1c9b0b0124405398afc3033537cf76" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46561 } } peers { permanent_uuid: "d3114496d23749e48a2889e0a3ceef40" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45757 } }
I20260806 02:52:08.370751 30035 raft_consensus.cc:374] T 62d24ea6f13d4492974018c1b869ca0f P b5272675298b46c4b9bea5414bd422b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:08.370857 30035 raft_consensus.cc:729] T 62d24ea6f13d4492974018c1b869ca0f P b5272675298b46c4b9bea5414bd422b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5272675298b46c4b9bea5414bd422b4, State: Initialized, Role: FOLLOWER
I20260806 02:52:08.371120 30035 consensus_queue.cc:260] T 62d24ea6f13d4492974018c1b869ca0f P b5272675298b46c4b9bea5414bd422b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5272675298b46c4b9bea5414bd422b4" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 33379 } } peers { permanent_uuid: "9d1c9b0b0124405398afc3033537cf76" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46561 } } peers { permanent_uuid: "d3114496d23749e48a2889e0a3ceef40" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45757 } }
I20260806 02:52:08.371740 30035 ts_tablet_manager.cc:1428] T 62d24ea6f13d4492974018c1b869ca0f P b5272675298b46c4b9bea5414bd422b4: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260806 02:52:08.502301 30052 raft_consensus.cc:482] T 62d24ea6f13d4492974018c1b869ca0f P d3114496d23749e48a2889e0a3ceef40 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:52:08.502571 30052 raft_consensus.cc:504] T 62d24ea6f13d4492974018c1b869ca0f P d3114496d23749e48a2889e0a3ceef40 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5272675298b46c4b9bea5414bd422b4" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 33379 } } peers { permanent_uuid: "9d1c9b0b0124405398afc3033537cf76" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46561 } } peers { permanent_uuid: "d3114496d23749e48a2889e0a3ceef40" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45757 } }
I20260806 02:52:08.503290 30052 leader_election.cc:288] T 62d24ea6f13d4492974018c1b869ca0f P d3114496d23749e48a2889e0a3ceef40 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5272675298b46c4b9bea5414bd422b4 (127.25.211.129:33379), 9d1c9b0b0124405398afc3033537cf76 (127.25.211.130:46561)
I20260806 02:52:08.504006 29408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62d24ea6f13d4492974018c1b869ca0f" candidate_uuid: "d3114496d23749e48a2889e0a3ceef40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5272675298b46c4b9bea5414bd422b4" is_pre_election: true
I20260806 02:52:08.504189 29613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62d24ea6f13d4492974018c1b869ca0f" candidate_uuid: "d3114496d23749e48a2889e0a3ceef40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d1c9b0b0124405398afc3033537cf76" is_pre_election: true
I20260806 02:52:08.504436 29613 raft_consensus.cc:2454] T 62d24ea6f13d4492974018c1b869ca0f P 9d1c9b0b0124405398afc3033537cf76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3114496d23749e48a2889e0a3ceef40 in term 0.
I20260806 02:52:08.504491 29408 raft_consensus.cc:2454] T 62d24ea6f13d4492974018c1b869ca0f P b5272675298b46c4b9bea5414bd422b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3114496d23749e48a2889e0a3ceef40 in term 0.
I20260806 02:52:08.504973 29709 leader_election.cc:302] T 62d24ea6f13d4492974018c1b869ca0f P d3114496d23749e48a2889e0a3ceef40 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d1c9b0b0124405398afc3033537cf76, d3114496d23749e48a2889e0a3ceef40; no voters:
I20260806 02:52:08.505455 30052 raft_consensus.cc:2781] T 62d24ea6f13d4492974018c1b869ca0f P d3114496d23749e48a2889e0a3ceef40 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:52:08.505563 30052 raft_consensus.cc:482] T 62d24ea6f13d4492974018c1b869ca0f P d3114496d23749e48a2889e0a3ceef40 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:52:08.505681 30052 raft_consensus.cc:3037] T 62d24ea6f13d4492974018c1b869ca0f P d3114496d23749e48a2889e0a3ceef40 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:08.508104 30052 raft_consensus.cc:504] T 62d24ea6f13d4492974018c1b869ca0f P d3114496d23749e48a2889e0a3ceef40 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5272675298b46c4b9bea5414bd422b4" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 33379 } } peers { permanent_uuid: "9d1c9b0b0124405398afc3033537cf76" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46561 } } peers { permanent_uuid: "d3114496d23749e48a2889e0a3ceef40" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45757 } }
I20260806 02:52:08.508778 30052 leader_election.cc:288] T 62d24ea6f13d4492974018c1b869ca0f P d3114496d23749e48a2889e0a3ceef40 [CANDIDATE]: Term 1 election: Requested vote from peers b5272675298b46c4b9bea5414bd422b4 (127.25.211.129:33379), 9d1c9b0b0124405398afc3033537cf76 (127.25.211.130:46561)
I20260806 02:52:08.509420 29613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62d24ea6f13d4492974018c1b869ca0f" candidate_uuid: "d3114496d23749e48a2889e0a3ceef40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d1c9b0b0124405398afc3033537cf76"
I20260806 02:52:08.509446 29408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62d24ea6f13d4492974018c1b869ca0f" candidate_uuid: "d3114496d23749e48a2889e0a3ceef40" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5272675298b46c4b9bea5414bd422b4"
I20260806 02:52:08.509763 29408 raft_consensus.cc:3037] T 62d24ea6f13d4492974018c1b869ca0f P b5272675298b46c4b9bea5414bd422b4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:08.509760 29613 raft_consensus.cc:3037] T 62d24ea6f13d4492974018c1b869ca0f P 9d1c9b0b0124405398afc3033537cf76 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:08.513003 29408 raft_consensus.cc:2454] T 62d24ea6f13d4492974018c1b869ca0f P b5272675298b46c4b9bea5414bd422b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3114496d23749e48a2889e0a3ceef40 in term 1.
I20260806 02:52:08.512995 29613 raft_consensus.cc:2454] T 62d24ea6f13d4492974018c1b869ca0f P 9d1c9b0b0124405398afc3033537cf76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3114496d23749e48a2889e0a3ceef40 in term 1.
I20260806 02:52:08.513463 29709 leader_election.cc:302] T 62d24ea6f13d4492974018c1b869ca0f P d3114496d23749e48a2889e0a3ceef40 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d1c9b0b0124405398afc3033537cf76, d3114496d23749e48a2889e0a3ceef40; no voters:
I20260806 02:52:08.513885 30052 raft_consensus.cc:2781] T 62d24ea6f13d4492974018c1b869ca0f P d3114496d23749e48a2889e0a3ceef40 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:52:08.514147 30052 raft_consensus.cc:686] T 62d24ea6f13d4492974018c1b869ca0f P d3114496d23749e48a2889e0a3ceef40 [term 1 LEADER]: Becoming Leader. State: Replica: d3114496d23749e48a2889e0a3ceef40, State: Running, Role: LEADER
I20260806 02:52:08.514479 30052 consensus_queue.cc:237] T 62d24ea6f13d4492974018c1b869ca0f P d3114496d23749e48a2889e0a3ceef40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5272675298b46c4b9bea5414bd422b4" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 33379 } } peers { permanent_uuid: "9d1c9b0b0124405398afc3033537cf76" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46561 } } peers { permanent_uuid: "d3114496d23749e48a2889e0a3ceef40" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45757 } }
I20260806 02:52:08.517735 29171 catalog_manager.cc:5495] T 62d24ea6f13d4492974018c1b869ca0f P d3114496d23749e48a2889e0a3ceef40 reported cstate change: term changed from 0 to 1, leader changed from <none> to d3114496d23749e48a2889e0a3ceef40 (127.25.211.131). New cstate: current_term: 1 leader_uuid: "d3114496d23749e48a2889e0a3ceef40" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5272675298b46c4b9bea5414bd422b4" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 33379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d1c9b0b0124405398afc3033537cf76" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3114496d23749e48a2889e0a3ceef40" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 45757 } health_report { overall_health: HEALTHY } } }
I20260806 02:52:08.577669 29408 raft_consensus.cc:1261] T 62d24ea6f13d4492974018c1b869ca0f P b5272675298b46c4b9bea5414bd422b4 [term 1 FOLLOWER]: Refusing update from remote peer d3114496d23749e48a2889e0a3ceef40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:52:08.578110 29613 raft_consensus.cc:1261] T 62d24ea6f13d4492974018c1b869ca0f P 9d1c9b0b0124405398afc3033537cf76 [term 1 FOLLOWER]: Refusing update from remote peer d3114496d23749e48a2889e0a3ceef40: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:52:08.578790 30052 consensus_queue.cc:1035] T 62d24ea6f13d4492974018c1b869ca0f P d3114496d23749e48a2889e0a3ceef40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d1c9b0b0124405398afc3033537cf76" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:52:08.579263 30068 consensus_queue.cc:1035] T 62d24ea6f13d4492974018c1b869ca0f P d3114496d23749e48a2889e0a3ceef40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5272675298b46c4b9bea5414bd422b4" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 33379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 02:52:08.603569 30001 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260806 02:52:10.208611 29170 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:52:10.208877 29170 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:52:10.213912 29170 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 50d796ee82c3472496eacc2910ede519: Sending DeleteTablet for 3 replicas of tablet 62d24ea6f13d4492974018c1b869ca0f
I20260806 02:52:10.214890 29380 tablet_service.cc:1514] Processing DeleteTablet for tablet 62d24ea6f13d4492974018c1b869ca0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:52:10 UTC) from {username='kudu', principal='kudu/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:52294
I20260806 02:52:10.214980 29782 tablet_service.cc:1514] Processing DeleteTablet for tablet 62d24ea6f13d4492974018c1b869ca0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:52:10 UTC) from {username='kudu', principal='kudu/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:60664
I20260806 02:52:10.214973 29576 tablet_service.cc:1514] Processing DeleteTablet for tablet 62d24ea6f13d4492974018c1b869ca0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:52:10 UTC) from {username='kudu', principal='kudu/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:41152
I20260806 02:52:10.215698 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 29276
I20260806 02:52:10.224910 30118 tablet_replica.cc:330] T 62d24ea6f13d4492974018c1b869ca0f P b5272675298b46c4b9bea5414bd422b4: stopping tablet replica
I20260806 02:52:10.225157 30118 raft_consensus.cc:2229] T 62d24ea6f13d4492974018c1b869ca0f P b5272675298b46c4b9bea5414bd422b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:52:10.225426 30118 raft_consensus.cc:2258] T 62d24ea6f13d4492974018c1b869ca0f P b5272675298b46c4b9bea5414bd422b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:52:10.227089 30119 tablet_replica.cc:330] T 62d24ea6f13d4492974018c1b869ca0f P d3114496d23749e48a2889e0a3ceef40: stopping tablet replica
I20260806 02:52:10.227314 30119 raft_consensus.cc:2229] T 62d24ea6f13d4492974018c1b869ca0f P d3114496d23749e48a2889e0a3ceef40 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:52:10.227787 30119 raft_consensus.cc:2258] T 62d24ea6f13d4492974018c1b869ca0f P d3114496d23749e48a2889e0a3ceef40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:52:10.228514 30118 ts_tablet_manager.cc:1905] T 62d24ea6f13d4492974018c1b869ca0f P b5272675298b46c4b9bea5414bd422b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:52:10.230902 30119 ts_tablet_manager.cc:1905] T 62d24ea6f13d4492974018c1b869ca0f P d3114496d23749e48a2889e0a3ceef40: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 06 02:52:10 dist-test-slave-zj8q krb5kdc[29106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984727, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.25.211.129@KRBTEST.COM
I20260806 02:52:10.241407 30122 tablet_replica.cc:330] T 62d24ea6f13d4492974018c1b869ca0f P 9d1c9b0b0124405398afc3033537cf76: stopping tablet replica
I20260806 02:52:10.241892 30119 ts_tablet_manager.cc:1918] T 62d24ea6f13d4492974018c1b869ca0f P d3114496d23749e48a2889e0a3ceef40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 02:52:10.242293 30122 raft_consensus.cc:2229] T 62d24ea6f13d4492974018c1b869ca0f P 9d1c9b0b0124405398afc3033537cf76 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:52:10.242558 30119 log.cc:1196] T 62d24ea6f13d4492974018c1b869ca0f P d3114496d23749e48a2889e0a3ceef40: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/ts-2/wal/wals/62d24ea6f13d4492974018c1b869ca0f
I20260806 02:52:10.242866 30122 raft_consensus.cc:2258] T 62d24ea6f13d4492974018c1b869ca0f P 9d1c9b0b0124405398afc3033537cf76 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:52:10.244499 30118 ts_tablet_manager.cc:1918] T 62d24ea6f13d4492974018c1b869ca0f P b5272675298b46c4b9bea5414bd422b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 02:52:10.244786 30118 log.cc:1196] T 62d24ea6f13d4492974018c1b869ca0f P b5272675298b46c4b9bea5414bd422b4: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/ts-0/wal/wals/62d24ea6f13d4492974018c1b869ca0f
I20260806 02:52:10.245659 30118 ts_tablet_manager.cc:1939] T 62d24ea6f13d4492974018c1b869ca0f P b5272675298b46c4b9bea5414bd422b4: Deleting consensus metadata
I20260806 02:52:10.247439 30122 ts_tablet_manager.cc:1905] T 62d24ea6f13d4492974018c1b869ca0f P 9d1c9b0b0124405398afc3033537cf76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:52:10.243477 30119 ts_tablet_manager.cc:1939] T 62d24ea6f13d4492974018c1b869ca0f P d3114496d23749e48a2889e0a3ceef40: Deleting consensus metadata
I20260806 02:52:10.247642 29151 catalog_manager.cc:4841] TS b5272675298b46c4b9bea5414bd422b4 (127.25.211.129:33379): tablet 62d24ea6f13d4492974018c1b869ca0f (table test-table [id=22e39ee0aea04808a1717a36e6314fab]) successfully deleted
I20260806 02:52:10.248595 30125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:10.217035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52324 (local address 127.25.211.129:33379)
0806 02:52:10.218633 (+ 1598us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:10.218640 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:10.219084 (+ 444us) server_negotiation.cc:407] Connection header received
0806 02:52:10.219355 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:10.219363 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:10.219461 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:10.219545 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:10.221924 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:10.223401 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:10.225546 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:10.226465 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:10.226583 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:10.238599 (+ 12016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:10.238623 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:10.238649 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:10.238701 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:10.242993 (+ 4292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:10.244108 (+ 1115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:10.244120 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:10.244123 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:10.244400 (+ 277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:10.245117 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:10.245123 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:10.245126 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:10.245707 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:10.246040 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:10.246966 (+ 926us) server_negotiation.cc:299] Negotiation successful
0806 02:52:10.247202 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1462,"thread_start_us":1309,"threads_started":1}
I20260806 02:52:10.249275 29150 catalog_manager.cc:4841] TS d3114496d23749e48a2889e0a3ceef40 (127.25.211.131:45757): tablet 62d24ea6f13d4492974018c1b869ca0f (table test-table [id=22e39ee0aea04808a1717a36e6314fab]) successfully deleted
W20260806 02:52:10.249604 29468 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52324
W20260806 02:52:10.250447 26446 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 29276: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 02:52:10.250774 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 29276
I20260806 02:52:10.256032 30122 ts_tablet_manager.cc:1918] T 62d24ea6f13d4492974018c1b869ca0f P 9d1c9b0b0124405398afc3033537cf76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 02:52:10.256263 30122 log.cc:1196] T 62d24ea6f13d4492974018c1b869ca0f P 9d1c9b0b0124405398afc3033537cf76: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1785984707115702-26446-0/minicluster-data/ts-1/wal/wals/62d24ea6f13d4492974018c1b869ca0f
I20260806 02:52:10.256716 30122 ts_tablet_manager.cc:1939] T 62d24ea6f13d4492974018c1b869ca0f P 9d1c9b0b0124405398afc3033537cf76: Deleting consensus metadata
I20260806 02:52:10.258131 29151 catalog_manager.cc:4841] TS 9d1c9b0b0124405398afc3033537cf76 (127.25.211.130:46561): tablet 62d24ea6f13d4492974018c1b869ca0f (table test-table [id=22e39ee0aea04808a1717a36e6314fab]) successfully deleted
I20260806 02:52:10.286520 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 29483
Aug 06 02:52:10 dist-test-slave-zj8q krb5kdc[29106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984727, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.25.211.130@KRBTEST.COM
I20260806 02:52:10.301766 30126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:10.287359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41168 (local address 127.25.211.130:46561)
0806 02:52:10.287892 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:10.287896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:10.287913 (+ 17us) server_negotiation.cc:407] Connection header received
0806 02:52:10.287969 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:10.287973 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:10.288031 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:10.288140 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:10.289452 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:10.290407 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:10.291611 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:10.291993 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:10.292097 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:10.296273 (+ 4176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:10.296289 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:10.296306 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:10.296342 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:10.299217 (+ 2875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:10.299860 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:10.299868 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:10.299870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:10.300007 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:10.300288 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:10.300291 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:10.300293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:10.300650 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:10.300890 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:10.301230 (+ 340us) server_negotiation.cc:299] Negotiation successful
0806 02:52:10.301418 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":262,"threads_started":1}
W20260806 02:52:10.302510 29675 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41168
W20260806 02:52:10.303126 26446 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 29483: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 02:52:10.303272 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 29483
I20260806 02:52:10.328976 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 29684
Aug 06 02:52:10 dist-test-slave-zj8q krb5kdc[29106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984727, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.25.211.131@KRBTEST.COM
I20260806 02:52:10.345928 30127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:10.329784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60690 (local address 127.25.211.131:45757)
0806 02:52:10.330364 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:10.330369 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:10.330385 (+ 16us) server_negotiation.cc:407] Connection header received
0806 02:52:10.330441 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:10.330445 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:10.330511 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:10.330607 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:10.331990 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:10.332780 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:10.334081 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:10.334598 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:10.334717 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:10.339185 (+ 4468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:10.339204 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:10.339231 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:10.339273 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:10.342760 (+ 3487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:10.343542 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:10.343551 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:10.343553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:10.343742 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:10.344229 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:10.344234 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:10.344236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:10.344727 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:10.344982 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:10.345341 (+ 359us) server_negotiation.cc:299] Negotiation successful
0806 02:52:10.345618 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":263,"threads_started":1}
W20260806 02:52:10.346673 29878 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60690
W20260806 02:52:10.347379 26446 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 29684: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 02:52:10.347549 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 29684
I20260806 02:52:10.377599 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 29122
I20260806 02:52:10.378858 29230 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:52:10.496472 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 29122
2026-08-06T02:52:10Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8978 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:52:10 dist-test-slave-zj8q krb5kdc[30132](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:52:10 dist-test-slave-zj8q krb5kdc[30132](info): set up 2 sockets
Aug 06 02:52:10 dist-test-slave-zj8q krb5kdc[30132](info): commencing operation
krb5kdc: starting...
W20260806 02:52:11.743639 26446 mini_kdc.cc:121] Time spent starting KDC: real 1.199s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260806 02:52:11.911266 26446 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.167s user 0.000s sys 0.008s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260806 02:52:12.109525 26446 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.198s user 0.000s sys 0.008s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260806 02:52:12.342944 26446 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.233s user 0.000s sys 0.007s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/krb5kdc/test-user.keytab.
W20260806 02:52:12.556697 26446 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.214s user 0.000s sys 0.008s
Aug 06 02:52:12 dist-test-slave-zj8q krb5kdc[30132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:52:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:52:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.211.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab.
Entry for principal kudu/127.25.211.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab.
I20260806 02:52:12.671698 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:37771
--webserver_interface=127.25.211.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:36123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.211.190:37771
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:52:13.018939 30149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:52:13.019240 30149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:52:13.019379 30149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:52:13.026842 30149 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:52:13.026965 30149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:52:13.027022 30149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:52:13.027099 30149 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:52:13.027153 30149 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:52:13.037531 30149 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:36123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.211.190:37771
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:37771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.211.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:52:13.039044 30149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:52:13.040442 30149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:52:13.052915 30154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:13.052845 30157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:13.052870 30155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:52:13.053277 30149 server_base.cc:1034] running on GCE node
I20260806 02:52:13.054417 30149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:52:13.055943 30149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:52:13.057191 30149 hybrid_clock.cc:648] HybridClock initialized: now 1785984733057129 us; error 39 us; skew 500 ppm
I20260806 02:52:13.057567 30149 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:52:13 dist-test-slave-zj8q krb5kdc[30132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984733, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:52:13.060907 30149 init.cc:377] Logged in from keytab as kudu/127.25.211.190@KRBTEST.COM (short username kudu)
I20260806 02:52:13.062646 30149 webserver.cc:460] Webserver started at http://127.25.211.190:45559/ using document root <none> and password file <none>
I20260806 02:52:13.063372 30149 fs_manager.cc:359] Metadata directory not provided
I20260806 02:52:13.063480 30149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:52:13.063824 30149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:52:13.067139 30149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/master-0/data/instance:
uuid: "b36094d8a5864001a755e1bd4147e426"
format_stamp: "Formatted at 2026-08-06 02:52:13 on dist-test-slave-zj8q"
server_key: "10dcae447262d51e5134fcdd2b697ac8"
server_key_iv: "522becda49e5d831645904f7f2540540"
server_key_version: "encryptionkey@0"
I20260806 02:52:13.067947 30149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/master-0/wal/instance:
uuid: "b36094d8a5864001a755e1bd4147e426"
format_stamp: "Formatted at 2026-08-06 02:52:13 on dist-test-slave-zj8q"
server_key: "10dcae447262d51e5134fcdd2b697ac8"
server_key_iv: "522becda49e5d831645904f7f2540540"
server_key_version: "encryptionkey@0"
I20260806 02:52:13.073764 30149 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260806 02:52:13.077690 30166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:13.079398 30149 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:52:13.079586 30149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/master-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/master-0/wal
uuid: "b36094d8a5864001a755e1bd4147e426"
format_stamp: "Formatted at 2026-08-06 02:52:13 on dist-test-slave-zj8q"
server_key: "10dcae447262d51e5134fcdd2b697ac8"
server_key_iv: "522becda49e5d831645904f7f2540540"
server_key_version: "encryptionkey@0"
I20260806 02:52:13.079831 30149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:52:13.103590 30149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:52:13.108304 30149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:52:13.108664 30149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:52:13.128476 30149 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.190:37771
I20260806 02:52:13.128482 30243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.190:37771 every 8 connection(s)
I20260806 02:52:13.130031 30149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
I20260806 02:52:13.134138 30244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:13.135612 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 30149
I20260806 02:52:13.135879 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/master-0/wal/instance
I20260806 02:52:13.136713 26446 external_mini_cluster.cc:1428] Setting key 3af6846e5848ff347b1ed6f7014350e2
I20260806 02:52:13.148361 30244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b36094d8a5864001a755e1bd4147e426: Bootstrap starting.
Aug 06 02:52:13 dist-test-slave-zj8q krb5kdc[30132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:52:13.151917 30244 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b36094d8a5864001a755e1bd4147e426: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:13.153093 30244 log.cc:824] T 00000000000000000000000000000000 P b36094d8a5864001a755e1bd4147e426: Log is configured to *not* fsync() on all Append() calls
I20260806 02:52:13.156653 30244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b36094d8a5864001a755e1bd4147e426: No bootstrap required, opened a new log
I20260806 02:52:13.161348 30250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:13.139099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54160 (local address 127.25.211.190:37771)
0806 02:52:13.139924 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:13.139936 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:13.139990 (+ 54us) server_negotiation.cc:407] Connection header received
0806 02:52:13.141658 (+ 1668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:13.141690 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:13.142239 (+ 549us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:13.142727 (+ 488us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:13.144408 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:13.145922 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:13.147251 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:13.147935 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:13.148130 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:13.152339 (+ 4209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:13.152352 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:13.152368 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:13.152414 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:13.156162 (+ 3748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:13.156913 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:13.156923 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:13.156932 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:13.157176 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:13.157651 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:13.157656 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:13.157659 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:13.158462 (+ 803us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:13.158795 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:13.159299 (+ 504us) server_negotiation.cc:299] Negotiation successful
0806 02:52:13.159815 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":598,"thread_start_us":333,"threads_started":1}
I20260806 02:52:13.166698 30244 raft_consensus.cc:348] T 00000000000000000000000000000000 P b36094d8a5864001a755e1bd4147e426 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b36094d8a5864001a755e1bd4147e426" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 37771 } }
I20260806 02:52:13.167105 30244 raft_consensus.cc:374] T 00000000000000000000000000000000 P b36094d8a5864001a755e1bd4147e426 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:13.167186 30244 raft_consensus.cc:729] T 00000000000000000000000000000000 P b36094d8a5864001a755e1bd4147e426 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b36094d8a5864001a755e1bd4147e426, State: Initialized, Role: FOLLOWER
I20260806 02:52:13.167726 30244 consensus_queue.cc:260] T 00000000000000000000000000000000 P b36094d8a5864001a755e1bd4147e426 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b36094d8a5864001a755e1bd4147e426" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 37771 } }
I20260806 02:52:13.167968 30244 raft_consensus.cc:388] T 00000000000000000000000000000000 P b36094d8a5864001a755e1bd4147e426 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:52:13.168069 30244 raft_consensus.cc:482] T 00000000000000000000000000000000 P b36094d8a5864001a755e1bd4147e426 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:52:13.168215 30244 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b36094d8a5864001a755e1bd4147e426 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:13.170720 30244 raft_consensus.cc:504] T 00000000000000000000000000000000 P b36094d8a5864001a755e1bd4147e426 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b36094d8a5864001a755e1bd4147e426" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 37771 } }
I20260806 02:52:13.171274 30244 leader_election.cc:302] T 00000000000000000000000000000000 P b36094d8a5864001a755e1bd4147e426 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b36094d8a5864001a755e1bd4147e426; no voters:
I20260806 02:52:13.171895 30244 leader_election.cc:288] T 00000000000000000000000000000000 P b36094d8a5864001a755e1bd4147e426 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:52:13.172035 30252 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b36094d8a5864001a755e1bd4147e426 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:52:13.172786 30252 raft_consensus.cc:686] T 00000000000000000000000000000000 P b36094d8a5864001a755e1bd4147e426 [term 1 LEADER]: Becoming Leader. State: Replica: b36094d8a5864001a755e1bd4147e426, State: Running, Role: LEADER
I20260806 02:52:13.173349 30252 consensus_queue.cc:237] T 00000000000000000000000000000000 P b36094d8a5864001a755e1bd4147e426 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b36094d8a5864001a755e1bd4147e426" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 37771 } }
I20260806 02:52:13.174090 30244 sys_catalog.cc:564] T 00000000000000000000000000000000 P b36094d8a5864001a755e1bd4147e426 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:52:13.176311 30255 sys_catalog.cc:455] T 00000000000000000000000000000000 P b36094d8a5864001a755e1bd4147e426 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b36094d8a5864001a755e1bd4147e426. Latest consensus state: current_term: 1 leader_uuid: "b36094d8a5864001a755e1bd4147e426" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b36094d8a5864001a755e1bd4147e426" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 37771 } } }
I20260806 02:52:13.176642 30255 sys_catalog.cc:458] T 00000000000000000000000000000000 P b36094d8a5864001a755e1bd4147e426 [sys.catalog]: This master's current role is: LEADER
I20260806 02:52:13.177490 30264 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:52:13.177349 30253 sys_catalog.cc:455] T 00000000000000000000000000000000 P b36094d8a5864001a755e1bd4147e426 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b36094d8a5864001a755e1bd4147e426" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b36094d8a5864001a755e1bd4147e426" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 37771 } } }
I20260806 02:52:13.178682 30253 sys_catalog.cc:458] T 00000000000000000000000000000000 P b36094d8a5864001a755e1bd4147e426 [sys.catalog]: This master's current role is: LEADER
I20260806 02:52:13.183300 30264 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:52:13.194509 30264 catalog_manager.cc:1347] Generated new cluster ID: 1d8cd6648b80416c9cc039e77c617759
I20260806 02:52:13.194667 30264 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:52:13.206032 30264 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:52:13.207520 30264 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:52:13.218689 30264 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b36094d8a5864001a755e1bd4147e426: Generated new TSK 0
I20260806 02:52:13.219913 30264 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.211.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab.
Entry for principal kudu/127.25.211.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab.
I20260806 02:52:13.277561 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.129:0
--local_ip_for_outbound_sockets=127.25.211.129
--webserver_interface=127.25.211.129
--webserver_port=0
--tserver_master_addrs=127.25.211.190:37771
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab
--principal=kudu/127.25.211.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:36123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:52:13.627851 30283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:52:13.628158 30283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:52:13.628255 30283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:52:13.635895 30283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:52:13.636032 30283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:52:13.636147 30283 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.129
I20260806 02:52:13.646458 30283 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:36123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab
--principal=kudu/127.25.211.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.211.129
--webserver_port=0
--tserver_master_addrs=127.25.211.190:37771
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.129
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:52:13.647800 30283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:52:13.649230 30283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:52:13.664887 30292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:13.665591 30291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:13.665704 30294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:52:13.666940 30283 server_base.cc:1034] running on GCE node
I20260806 02:52:13.667694 30283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:52:13.668793 30283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:52:13.670038 30283 hybrid_clock.cc:648] HybridClock initialized: now 1785984733669968 us; error 51 us; skew 500 ppm
I20260806 02:52:13.670459 30283 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:52:13 dist-test-slave-zj8q krb5kdc[30132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984733, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:52:13.673974 30283 init.cc:377] Logged in from keytab as kudu/127.25.211.129@KRBTEST.COM (short username kudu)
I20260806 02:52:13.675791 30283 webserver.cc:460] Webserver started at http://127.25.211.129:44233/ using document root <none> and password file <none>
I20260806 02:52:13.676538 30283 fs_manager.cc:359] Metadata directory not provided
I20260806 02:52:13.676671 30283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:52:13.676994 30283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:52:13.680506 30283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-0/data/instance:
uuid: "ec13b33e144b42739f8c242634977e5d"
format_stamp: "Formatted at 2026-08-06 02:52:13 on dist-test-slave-zj8q"
server_key: "9caa0a2b180c1a993e4e4d720555fb34"
server_key_iv: "c06ceea6017095921ca2cb0230fb1027"
server_key_version: "encryptionkey@0"
I20260806 02:52:13.681408 30283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-0/wal/instance:
uuid: "ec13b33e144b42739f8c242634977e5d"
format_stamp: "Formatted at 2026-08-06 02:52:13 on dist-test-slave-zj8q"
server_key: "9caa0a2b180c1a993e4e4d720555fb34"
server_key_iv: "c06ceea6017095921ca2cb0230fb1027"
server_key_version: "encryptionkey@0"
I20260806 02:52:13.687569 30283 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260806 02:52:13.691732 30303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:13.693331 30283 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:52:13.693518 30283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-0/wal
uuid: "ec13b33e144b42739f8c242634977e5d"
format_stamp: "Formatted at 2026-08-06 02:52:13 on dist-test-slave-zj8q"
server_key: "9caa0a2b180c1a993e4e4d720555fb34"
server_key_iv: "c06ceea6017095921ca2cb0230fb1027"
server_key_version: "encryptionkey@0"
I20260806 02:52:13.693750 30283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:52:13.720745 30283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:52:13.725415 30283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:52:13.725708 30283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:52:13.726792 30283 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:52:13.728242 30283 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:52:13.728330 30283 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:13.728451 30283 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:52:13.728502 30283 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:13.759387 30283 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.129:45193
I20260806 02:52:13.759429 30455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.129:45193 every 8 connection(s)
I20260806 02:52:13.760773 30283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
I20260806 02:52:13.762312 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 30283
I20260806 02:52:13.762672 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-0/wal/instance
I20260806 02:52:13.763732 26446 external_mini_cluster.cc:1428] Setting key b6802001322630b3146467582f7fd11e
Aug 06 02:52:13 dist-test-slave-zj8q krb5kdc[30132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984733, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.129@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:52:13.783638 30458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:13.763752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:58555 (local address 127.25.211.190:37771)
0806 02:52:13.766273 (+ 2521us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:13.766279 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:13.766299 (+ 20us) server_negotiation.cc:407] Connection header received
0806 02:52:13.768069 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:13.768076 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:13.768161 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:13.768281 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:13.770018 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:13.771052 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:13.772718 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:13.773270 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:13.773408 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:13.777386 (+ 3978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:13.777392 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:13.777395 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:13.777431 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:13.780191 (+ 2760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:13.781009 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:13.781013 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:13.781016 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:13.781157 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:13.781616 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:13.781620 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:13.781622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:13.781926 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:13.782068 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:13.782960 (+ 892us) server_negotiation.cc:299] Negotiation successful
0806 02:52:13.783223 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2374,"thread_start_us":2230,"threads_started":1}
I20260806 02:52:13.784755 30459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:13.764506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:37771 (local address 127.25.211.129:58555)
0806 02:52:13.765614 (+ 1108us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:13.765693 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:13.767743 (+ 2050us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:13.768591 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:13.768600 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:13.769186 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:13.769825 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:13.769844 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:13.771222 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:13.771226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:13.772560 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:13.772575 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:13.773521 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:13.773527 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:13.773779 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:13.774550 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:52:13.774575 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:13.777130 (+ 2555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:13.780374 (+ 3244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:13.780385 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:13.780401 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:13.780875 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:13.781276 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:13.781280 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:13.781283 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:13.781493 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:13.782088 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:13.782098 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:13.782551 (+ 453us) client_negotiation.cc:769] Sending connection context
0806 02:52:13.782820 (+ 269us) client_negotiation.cc:240] Negotiation successful
0806 02:52:13.783450 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":738,"thread_start_us":476,"threads_started":1}
WARNING: no policy specified for kudu/127.25.211.130@KRBTEST.COM; defaulting to no policy
I20260806 02:52:13.786763 30456 heartbeater.cc:347] Connected to a master server at 127.25.211.190:37771
I20260806 02:52:13.787122 30456 heartbeater.cc:464] Registering TS with master...
I20260806 02:52:13.787923 30456 heartbeater.cc:511] Master 127.25.211.190:37771 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.130@KRBTEST.COM" created.
I20260806 02:52:13.790365 30188 ts_manager.cc:194] Registered new tserver with Master: ec13b33e144b42739f8c242634977e5d (127.25.211.129:45193)
I20260806 02:52:13.791858 30188 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.211.129@KRBTEST.COM'} at 127.25.211.129:58555
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab.
Entry for principal kudu/127.25.211.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab.
I20260806 02:52:13.817395 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.130:0
--local_ip_for_outbound_sockets=127.25.211.130
--webserver_interface=127.25.211.130
--webserver_port=0
--tserver_master_addrs=127.25.211.190:37771
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab
--principal=kudu/127.25.211.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:36123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:52:14.175895 30465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:52:14.176224 30465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:52:14.176330 30465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:52:14.184082 30465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:52:14.184234 30465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:52:14.184371 30465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.130
I20260806 02:52:14.194890 30465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:36123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab
--principal=kudu/127.25.211.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.211.130
--webserver_port=0
--tserver_master_addrs=127.25.211.190:37771
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.130
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:52:14.196211 30465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:52:14.197830 30465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:52:14.211737 30474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:14.211704 30475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:14.211773 30480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:52:14.213044 30465 server_base.cc:1034] running on GCE node
I20260806 02:52:14.213835 30465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:52:14.214948 30465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:52:14.216193 30465 hybrid_clock.cc:648] HybridClock initialized: now 1785984734216130 us; error 53 us; skew 500 ppm
I20260806 02:52:14.216568 30465 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:52:14 dist-test-slave-zj8q krb5kdc[30132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984734, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:52:14.220219 30465 init.cc:377] Logged in from keytab as kudu/127.25.211.130@KRBTEST.COM (short username kudu)
I20260806 02:52:14.221938 30465 webserver.cc:460] Webserver started at http://127.25.211.130:36207/ using document root <none> and password file <none>
I20260806 02:52:14.222725 30465 fs_manager.cc:359] Metadata directory not provided
I20260806 02:52:14.222853 30465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:52:14.223181 30465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:52:14.226346 30465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-1/data/instance:
uuid: "78c09185fa2b4770ad36b2f6673b4548"
format_stamp: "Formatted at 2026-08-06 02:52:14 on dist-test-slave-zj8q"
server_key: "2f8f0687e3b4a75911f7eef0dcb3bebd"
server_key_iv: "9fa7b2aea3455d68e4f5852376dedd46"
server_key_version: "encryptionkey@0"
I20260806 02:52:14.227198 30465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-1/wal/instance:
uuid: "78c09185fa2b4770ad36b2f6673b4548"
format_stamp: "Formatted at 2026-08-06 02:52:14 on dist-test-slave-zj8q"
server_key: "2f8f0687e3b4a75911f7eef0dcb3bebd"
server_key_iv: "9fa7b2aea3455d68e4f5852376dedd46"
server_key_version: "encryptionkey@0"
I20260806 02:52:14.232844 30465 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260806 02:52:14.236824 30488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:14.238373 30465 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260806 02:52:14.238550 30465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-1/wal
uuid: "78c09185fa2b4770ad36b2f6673b4548"
format_stamp: "Formatted at 2026-08-06 02:52:14 on dist-test-slave-zj8q"
server_key: "2f8f0687e3b4a75911f7eef0dcb3bebd"
server_key_iv: "9fa7b2aea3455d68e4f5852376dedd46"
server_key_version: "encryptionkey@0"
I20260806 02:52:14.238770 30465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:52:14.252704 30465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:52:14.257581 30465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:52:14.257910 30465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:52:14.259038 30465 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:52:14.260524 30465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:52:14.260605 30465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:14.260725 30465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:52:14.260804 30465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:14.293561 30465 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.130:42121
I20260806 02:52:14.293603 30638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.130:42121 every 8 connection(s)
I20260806 02:52:14.294921 30465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
I20260806 02:52:14.303125 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 30465
I20260806 02:52:14.303354 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-1/wal/instance
I20260806 02:52:14.303815 26446 external_mini_cluster.cc:1428] Setting key 05a52cadc99e8d733bddc4daf6999497
Aug 06 02:52:14 dist-test-slave-zj8q krb5kdc[30132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984734, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.130@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:52:14.318127 30641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:14.297515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:41147 (local address 127.25.211.190:37771)
0806 02:52:14.297952 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:14.297957 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:14.299048 (+ 1091us) server_negotiation.cc:407] Connection header received
0806 02:52:14.301024 (+ 1976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:14.301029 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:14.301107 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:14.301208 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:14.303186 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:14.304103 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:14.306277 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:14.306759 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:14.306864 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:14.311333 (+ 4469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:14.311342 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:14.311347 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:14.311397 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:14.314624 (+ 3227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:14.315752 (+ 1128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:14.315757 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:14.315760 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:14.315906 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:14.316391 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:14.316394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:14.316396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:14.316675 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:14.316788 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:14.317670 (+ 882us) server_negotiation.cc:299] Negotiation successful
0806 02:52:14.317857 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":214,"threads_started":1}
I20260806 02:52:14.319823 30643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:14.298068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:37771 (local address 127.25.211.130:41147)
0806 02:52:14.298876 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:14.298917 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:14.300681 (+ 1764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:14.301451 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:14.301460 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:14.302071 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:14.302922 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:14.302959 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:14.304313 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:14.304319 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:14.306093 (+ 1774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:14.306113 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:14.307024 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:14.307031 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:14.307349 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:14.308165 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:52:14.308200 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:14.310971 (+ 2771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:14.314826 (+ 3855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:14.314839 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:14.314862 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:14.315583 (+ 721us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:14.316022 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:14.316027 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:14.316030 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:14.316292 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:14.316797 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:14.316805 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:14.317273 (+ 468us) client_negotiation.cc:769] Sending connection context
0806 02:52:14.317594 (+ 321us) client_negotiation.cc:240] Negotiation successful
0806 02:52:14.318213 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":300,"threads_started":1}
I20260806 02:52:14.321491 30639 heartbeater.cc:347] Connected to a master server at 127.25.211.190:37771
I20260806 02:52:14.321844 30639 heartbeater.cc:464] Registering TS with master...
I20260806 02:52:14.322680 30639 heartbeater.cc:511] Master 127.25.211.190:37771 requested a full tablet report, sending...
I20260806 02:52:14.324331 30188 ts_manager.cc:194] Registered new tserver with Master: 78c09185fa2b4770ad36b2f6673b4548 (127.25.211.130:42121)
I20260806 02:52:14.325165 30188 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.211.130@KRBTEST.COM'} at 127.25.211.130:41147
WARNING: no policy specified for kudu/127.25.211.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab.
Entry for principal kudu/127.25.211.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab.
I20260806 02:52:14.355643 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.131:0
--local_ip_for_outbound_sockets=127.25.211.131
--webserver_interface=127.25.211.131
--webserver_port=0
--tserver_master_addrs=127.25.211.190:37771
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab
--principal=kudu/127.25.211.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:36123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:52:14.697096 30647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:52:14.697443 30647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:52:14.697576 30647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:52:14.705150 30647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:52:14.705304 30647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:52:14.705487 30647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.131
I20260806 02:52:14.716346 30647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:36123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab
--principal=kudu/127.25.211.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.211.131
--webserver_port=0
--tserver_master_addrs=127.25.211.190:37771
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.131
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:52:14.717808 30647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:52:14.719262 30647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:52:14.732757 30652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:14.732789 30657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:14.732717 30654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:52:14.733520 30647 server_base.cc:1034] running on GCE node
I20260806 02:52:14.734253 30647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:52:14.735284 30647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:52:14.736495 30647 hybrid_clock.cc:648] HybridClock initialized: now 1785984734736440 us; error 48 us; skew 500 ppm
I20260806 02:52:14.736831 30647 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:52:14 dist-test-slave-zj8q krb5kdc[30132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984734, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:52:14.740046 30647 init.cc:377] Logged in from keytab as kudu/127.25.211.131@KRBTEST.COM (short username kudu)
I20260806 02:52:14.741600 30647 webserver.cc:460] Webserver started at http://127.25.211.131:35023/ using document root <none> and password file <none>
I20260806 02:52:14.742410 30647 fs_manager.cc:359] Metadata directory not provided
I20260806 02:52:14.742554 30647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:52:14.742888 30647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:52:14.746091 30647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-2/data/instance:
uuid: "295c31768aaa445a983bdceab85d7b28"
format_stamp: "Formatted at 2026-08-06 02:52:14 on dist-test-slave-zj8q"
server_key: "7c8ad42b56c986a11ad9794855c4c18f"
server_key_iv: "13622afcd9d86debac16215fa59380b4"
server_key_version: "encryptionkey@0"
I20260806 02:52:14.747042 30647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-2/wal/instance:
uuid: "295c31768aaa445a983bdceab85d7b28"
format_stamp: "Formatted at 2026-08-06 02:52:14 on dist-test-slave-zj8q"
server_key: "7c8ad42b56c986a11ad9794855c4c18f"
server_key_iv: "13622afcd9d86debac16215fa59380b4"
server_key_version: "encryptionkey@0"
I20260806 02:52:14.752705 30647 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260806 02:52:14.756561 30665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:14.758100 30647 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:52:14.758314 30647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-2/wal
uuid: "295c31768aaa445a983bdceab85d7b28"
format_stamp: "Formatted at 2026-08-06 02:52:14 on dist-test-slave-zj8q"
server_key: "7c8ad42b56c986a11ad9794855c4c18f"
server_key_iv: "13622afcd9d86debac16215fa59380b4"
server_key_version: "encryptionkey@0"
I20260806 02:52:14.758518 30647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:52:14.781322 30647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:52:14.785809 30647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:52:14.786096 30647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:52:14.787153 30647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:52:14.788582 30647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:52:14.788667 30647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:14.788798 30647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:52:14.788853 30647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:14.795032 30456 heartbeater.cc:503] Master 127.25.211.190:37771 was elected leader, sending a full tablet report...
I20260806 02:52:14.821437 30647 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.131:44471
I20260806 02:52:14.821480 30823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.131:44471 every 8 connection(s)
I20260806 02:52:14.822857 30647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
I20260806 02:52:14.829001 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 30647
I20260806 02:52:14.829210 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-2/wal/instance
I20260806 02:52:14.829775 26446 external_mini_cluster.cc:1428] Setting key 56a0fe017ce3ac8b30f353627feeeba5
Aug 06 02:52:14 dist-test-slave-zj8q krb5kdc[30132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984734, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.131@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:52:14.843945 30826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:14.825437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:37469 (local address 127.25.211.190:37771)
0806 02:52:14.825873 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:14.825878 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:14.826818 (+ 940us) server_negotiation.cc:407] Connection header received
0806 02:52:14.828638 (+ 1820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:14.828644 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:14.828740 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:14.828846 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:14.831156 (+ 2310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:14.832141 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:14.833947 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:14.834416 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:14.834524 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:14.838204 (+ 3680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:14.838210 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:14.838214 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:14.838251 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:14.840718 (+ 2467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:14.841502 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:14.841506 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:14.841508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:14.841633 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:14.842044 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:14.842048 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:14.842050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:14.842412 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:14.842542 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:14.843369 (+ 827us) server_negotiation.cc:299] Negotiation successful
0806 02:52:14.843560 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":232,"threads_started":1}
I20260806 02:52:14.845250 30827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:14.825902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:37771 (local address 127.25.211.131:37469)
0806 02:52:14.826653 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:14.826690 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:14.828372 (+ 1682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:14.829272 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:14.829285 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:14.830063 (+ 778us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:14.830930 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:14.830959 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:14.832301 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:14.832306 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:14.833785 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:14.833800 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:14.834609 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:14.834615 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:14.834824 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:14.835519 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:52:14.835543 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:14.837932 (+ 2389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:14.840895 (+ 2963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:14.840904 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:14.840919 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:14.841396 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:14.841737 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:14.841741 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:14.841743 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:14.841949 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:14.842533 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:14.842540 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:14.842970 (+ 430us) client_negotiation.cc:769] Sending connection context
0806 02:52:14.843242 (+ 272us) client_negotiation.cc:240] Negotiation successful
0806 02:52:14.843813 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":300,"threads_started":1}
I20260806 02:52:14.846659 30824 heartbeater.cc:347] Connected to a master server at 127.25.211.190:37771
I20260806 02:52:14.846953 30824 heartbeater.cc:464] Registering TS with master...
I20260806 02:52:14.847679 30824 heartbeater.cc:511] Master 127.25.211.190:37771 requested a full tablet report, sending...
I20260806 02:52:14.849104 30188 ts_manager.cc:194] Registered new tserver with Master: 295c31768aaa445a983bdceab85d7b28 (127.25.211.131:44471)
I20260806 02:52:14.849897 30188 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.211.131@KRBTEST.COM'} at 127.25.211.131:37469
I20260806 02:52:14.858779 26446 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 02:52:14.900645 30826 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:52:14.897364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54166 (local address 127.25.211.190:37771)
0806 02:52:14.897805 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:14.897811 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:14.898388 (+ 577us) server_negotiation.cc:407] Connection header received
0806 02:52:14.898728 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:14.898735 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:14.898843 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:14.898971 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:14.900440 (+ 1469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54166: BlockingRecv error: recv got EOF from 127.0.0.1:54166 (error 108)
Metrics: {"server-negotiator.queue_time_us":223}
I20260806 02:52:14.903946 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 30283
W20260806 02:52:14.907243 26446 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 30283: Not authorized: Client connection negotiation failed: client connection to 127.25.211.129:45193: server requires authentication, but client does not have Kerberos credentials available
I20260806 02:52:14.907426 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 30283
I20260806 02:52:14.929523 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 30465
W20260806 02:52:14.932444 26446 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 30465: Not authorized: Client connection negotiation failed: client connection to 127.25.211.130:42121: server requires authentication, but client does not have Kerberos credentials available
I20260806 02:52:14.932636 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 30465
W20260806 02:52:14.932734 30839 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:52:14.930506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41492 (local address 127.25.211.130:42121)
0806 02:52:14.931078 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:14.931085 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:14.931111 (+ 26us) server_negotiation.cc:407] Connection header received
0806 02:52:14.931174 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:14.931180 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:14.931415 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:14.931572 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:14.932408 (+ 836us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41492: BlockingRecv error: recv got EOF from 127.0.0.1:41492 (error 108)
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":309,"threads_started":1}
I20260806 02:52:14.954137 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 30647
W20260806 02:52:14.957444 26446 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 30647: Not authorized: Client connection negotiation failed: client connection to 127.25.211.131:44471: server requires authentication, but client does not have Kerberos credentials available
I20260806 02:52:14.957592 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 30647
I20260806 02:52:14.979851 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 30149
I20260806 02:52:14.981237 30237 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:52:15.097095 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 30149
2026-08-06T02:52:15Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (4598 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984707115702-26446-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:52:15 dist-test-slave-zj8q krb5kdc[30849](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:52:15 dist-test-slave-zj8q krb5kdc[30849](info): set up 2 sockets
Aug 06 02:52:15 dist-test-slave-zj8q krb5kdc[30849](info): commencing operation
krb5kdc: starting...
W20260806 02:52:16.301268 26446 mini_kdc.cc:121] Time spent starting KDC: real 1.158s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Aug 06 02:52:16 dist-test-slave-zj8q krb5kdc[30849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:52:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:52:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.211.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab.
Entry for principal kudu/127.25.211.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab.
I20260806 02:52:16.535703 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984707115702-26446-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984707115702-26446-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984707115702-26446-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:41245
--webserver_interface=127.25.211.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:36823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.211.190:41245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:52:16.885598 30867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:52:16.885939 30867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:52:16.886077 30867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:52:16.893414 30867 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:52:16.893543 30867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:52:16.893632 30867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:52:16.893714 30867 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:52:16.893779 30867 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:52:16.904065 30867 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:36823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984707115702-26446-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984707115702-26446-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.211.190:41245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:41245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.211.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984707115702-26446-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:52:16.905475 30867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:52:16.906883 30867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:52:16.919039 30873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:16.919009 30875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:16.919436 30879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:52:16.920302 30867 server_base.cc:1034] running on GCE node
I20260806 02:52:16.921129 30867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:52:16.922657 30867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:52:16.923875 30867 hybrid_clock.cc:648] HybridClock initialized: now 1785984736923821 us; error 50 us; skew 500 ppm
I20260806 02:52:16.924242 30867 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:52:16 dist-test-slave-zj8q krb5kdc[30849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984736, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:52:16.938674 30867 init.cc:377] Logged in from keytab as kudu/127.25.211.190@KRBTEST.COM (short username kudu)
I20260806 02:52:16.940500 30867 webserver.cc:460] Webserver started at http://127.25.211.190:38491/ using document root <none> and password file <none>
I20260806 02:52:16.941269 30867 fs_manager.cc:359] Metadata directory not provided
I20260806 02:52:16.941404 30867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:52:16.941743 30867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:52:16.945024 30867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984707115702-26446-0/minicluster-data/master-0/data/instance:
uuid: "a45b2db9678a4b33bdf6c48af1cb1755"
format_stamp: "Formatted at 2026-08-06 02:52:16 on dist-test-slave-zj8q"
server_key: "65d565ccdecb94a2bc0e419e5fd28324"
server_key_iv: "428b71afd0c284804a60943405263e9d"
server_key_version: "encryptionkey@0"
I20260806 02:52:16.945869 30867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984707115702-26446-0/minicluster-data/master-0/wal/instance:
uuid: "a45b2db9678a4b33bdf6c48af1cb1755"
format_stamp: "Formatted at 2026-08-06 02:52:16 on dist-test-slave-zj8q"
server_key: "65d565ccdecb94a2bc0e419e5fd28324"
server_key_iv: "428b71afd0c284804a60943405263e9d"
server_key_version: "encryptionkey@0"
I20260806 02:52:16.951366 30867 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260806 02:52:16.954993 30890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:16.956369 30867 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:52:16.956542 30867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984707115702-26446-0/minicluster-data/master-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984707115702-26446-0/minicluster-data/master-0/wal
uuid: "a45b2db9678a4b33bdf6c48af1cb1755"
format_stamp: "Formatted at 2026-08-06 02:52:16 on dist-test-slave-zj8q"
server_key: "65d565ccdecb94a2bc0e419e5fd28324"
server_key_iv: "428b71afd0c284804a60943405263e9d"
server_key_version: "encryptionkey@0"
I20260806 02:52:16.956754 30867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984707115702-26446-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984707115702-26446-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984707115702-26446-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:52:16.978073 30867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:52:16.982411 30867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:52:16.982676 30867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:52:17.001058 30867 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.190:41245
I20260806 02:52:17.001122 30967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.190:41245 every 8 connection(s)
I20260806 02:52:17.002519 30867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
I20260806 02:52:17.006573 30968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:17.009275 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 30867
I20260806 02:52:17.009505 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1785984707115702-26446-0/minicluster-data/master-0/wal/instance
I20260806 02:52:17.010244 26446 external_mini_cluster.cc:1428] Setting key 4fff4fe6f4e1be8896246bb475f8a90e
I20260806 02:52:17.020524 30968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a45b2db9678a4b33bdf6c48af1cb1755: Bootstrap starting.
I20260806 02:52:17.024511 30968 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a45b2db9678a4b33bdf6c48af1cb1755: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:17.025657 30968 log.cc:824] T 00000000000000000000000000000000 P a45b2db9678a4b33bdf6c48af1cb1755: Log is configured to *not* fsync() on all Append() calls
Aug 06 02:52:17 dist-test-slave-zj8q krb5kdc[30849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984736, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:52:17.029142 30968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a45b2db9678a4b33bdf6c48af1cb1755: No bootstrap required, opened a new log
I20260806 02:52:17.037333 30971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:17.012549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50028 (local address 127.25.211.190:41245)
0806 02:52:17.013485 (+ 936us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:17.013502 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:17.013557 (+ 55us) server_negotiation.cc:407] Connection header received
0806 02:52:17.015923 (+ 2366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:17.015960 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:17.016590 (+ 630us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:17.017115 (+ 525us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:17.018700 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:17.020735 (+ 2035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:17.022129 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:17.023263 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:17.023453 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:17.027817 (+ 4364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:17.027831 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:17.027852 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:17.027905 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:17.032148 (+ 4243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:17.033043 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:17.033053 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:17.033087 (+ 34us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:17.033332 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:17.033717 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:17.033723 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:17.033727 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:17.034558 (+ 831us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:17.034868 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:17.035298 (+ 430us) server_negotiation.cc:299] Negotiation successful
0806 02:52:17.035819 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":396,"threads_started":1}
I20260806 02:52:17.039520 30968 raft_consensus.cc:348] T 00000000000000000000000000000000 P a45b2db9678a4b33bdf6c48af1cb1755 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a45b2db9678a4b33bdf6c48af1cb1755" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 41245 } }
I20260806 02:52:17.039902 30968 raft_consensus.cc:374] T 00000000000000000000000000000000 P a45b2db9678a4b33bdf6c48af1cb1755 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:17.040063 30968 raft_consensus.cc:729] T 00000000000000000000000000000000 P a45b2db9678a4b33bdf6c48af1cb1755 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a45b2db9678a4b33bdf6c48af1cb1755, State: Initialized, Role: FOLLOWER
I20260806 02:52:17.040732 30968 consensus_queue.cc:260] T 00000000000000000000000000000000 P a45b2db9678a4b33bdf6c48af1cb1755 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a45b2db9678a4b33bdf6c48af1cb1755" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 41245 } }
I20260806 02:52:17.040980 30968 raft_consensus.cc:388] T 00000000000000000000000000000000 P a45b2db9678a4b33bdf6c48af1cb1755 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:52:17.041139 30968 raft_consensus.cc:482] T 00000000000000000000000000000000 P a45b2db9678a4b33bdf6c48af1cb1755 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:52:17.041334 30968 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a45b2db9678a4b33bdf6c48af1cb1755 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:17.041864 26446 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:52:17.044601 30968 raft_consensus.cc:504] T 00000000000000000000000000000000 P a45b2db9678a4b33bdf6c48af1cb1755 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a45b2db9678a4b33bdf6c48af1cb1755" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 41245 } }
I20260806 02:52:17.045288 30968 leader_election.cc:302] T 00000000000000000000000000000000 P a45b2db9678a4b33bdf6c48af1cb1755 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a45b2db9678a4b33bdf6c48af1cb1755; no voters:
I20260806 02:52:17.046047 30968 leader_election.cc:288] T 00000000000000000000000000000000 P a45b2db9678a4b33bdf6c48af1cb1755 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:52:17.046217 30973 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a45b2db9678a4b33bdf6c48af1cb1755 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:52:17.047390 30973 raft_consensus.cc:686] T 00000000000000000000000000000000 P a45b2db9678a4b33bdf6c48af1cb1755 [term 1 LEADER]: Becoming Leader. State: Replica: a45b2db9678a4b33bdf6c48af1cb1755, State: Running, Role: LEADER
I20260806 02:52:17.048054 30973 consensus_queue.cc:237] T 00000000000000000000000000000000 P a45b2db9678a4b33bdf6c48af1cb1755 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a45b2db9678a4b33bdf6c48af1cb1755" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 41245 } }
I20260806 02:52:17.048522 30968 sys_catalog.cc:564] T 00000000000000000000000000000000 P a45b2db9678a4b33bdf6c48af1cb1755 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:52:17.051096 30975 sys_catalog.cc:455] T 00000000000000000000000000000000 P a45b2db9678a4b33bdf6c48af1cb1755 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a45b2db9678a4b33bdf6c48af1cb1755. Latest consensus state: current_term: 1 leader_uuid: "a45b2db9678a4b33bdf6c48af1cb1755" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a45b2db9678a4b33bdf6c48af1cb1755" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 41245 } } }
I20260806 02:52:17.051595 30975 sys_catalog.cc:458] T 00000000000000000000000000000000 P a45b2db9678a4b33bdf6c48af1cb1755 [sys.catalog]: This master's current role is: LEADER
I20260806 02:52:17.051137 30974 sys_catalog.cc:455] T 00000000000000000000000000000000 P a45b2db9678a4b33bdf6c48af1cb1755 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a45b2db9678a4b33bdf6c48af1cb1755" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a45b2db9678a4b33bdf6c48af1cb1755" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 41245 } } }
I20260806 02:52:17.052372 30974 sys_catalog.cc:458] T 00000000000000000000000000000000 P a45b2db9678a4b33bdf6c48af1cb1755 [sys.catalog]: This master's current role is: LEADER
I20260806 02:52:17.055084 30983 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:52:17.060410 30983 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:52:17.071619 30983 catalog_manager.cc:1347] Generated new cluster ID: f4b02b8218eb414ab77bb0e970c7f7df
I20260806 02:52:17.071762 30983 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:52:17.088593 30997 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 02:52:17.105477 30983 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:52:17.107388 30983 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:52:17.112010 30971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:17.096360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50032 (local address 127.25.211.190:41245)
0806 02:52:17.096586 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:17.096592 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:17.102858 (+ 6266us) server_negotiation.cc:407] Connection header received
0806 02:52:17.103140 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:17.103147 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:17.103248 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:17.103378 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:17.104706 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:17.105915 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:17.107870 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:17.108589 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:17.108755 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:17.110078 (+ 1323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:17.110087 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:17.110092 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:52:17.111052 (+ 960us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:17.111141 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:52:17.111180 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:17.111307 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:17.111463 (+ 156us) server_negotiation.cc:299] Negotiation successful
0806 02:52:17.111627 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 02:52:17.125247 30983 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a45b2db9678a4b33bdf6c48af1cb1755: Generated new TSK 0
I20260806 02:52:17.126494 30983 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:52:17.139922 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 30867
I20260806 02:52:17.141214 30958 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:52:17.234169 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 30867
2026-08-06T02:52:17Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2131 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:52:17 dist-test-slave-zj8q krb5kdc[31013](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:52:17 dist-test-slave-zj8q krb5kdc[31013](info): set up 2 sockets
Aug 06 02:52:17 dist-test-slave-zj8q krb5kdc[31013](info): commencing operation
krb5kdc: starting...
W20260806 02:52:18.432649 26446 mini_kdc.cc:121] Time spent starting KDC: real 1.158s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Aug 06 02:52:18 dist-test-slave-zj8q krb5kdc[31013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:52:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:52:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.211.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab.
Entry for principal kudu/127.25.211.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab.
I20260806 02:52:18.670627 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:37009
--webserver_interface=127.25.211.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:46693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.211.190:37009
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:52:19.014974 31028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:52:19.015303 31028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:52:19.015455 31028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:52:19.022899 31028 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:52:19.023033 31028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:52:19.023092 31028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:52:19.023170 31028 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:52:19.023236 31028 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:52:19.033571 31028 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:46693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.211.190:37009
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:37009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.211.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:52:19.035125 31028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:52:19.036492 31028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:52:19.048542 31033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:19.048516 31037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:19.048808 31034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:52:19.049292 31028 server_base.cc:1034] running on GCE node
I20260806 02:52:19.050206 31028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:52:19.051764 31028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:52:19.052983 31028 hybrid_clock.cc:648] HybridClock initialized: now 1785984739052929 us; error 42 us; skew 500 ppm
I20260806 02:52:19.053347 31028 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:52:19 dist-test-slave-zj8q krb5kdc[31013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984739, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:52:19.056492 31028 init.cc:377] Logged in from keytab as kudu/127.25.211.190@KRBTEST.COM (short username kudu)
I20260806 02:52:19.057948 31028 webserver.cc:460] Webserver started at http://127.25.211.190:44021/ using document root <none> and password file <none>
I20260806 02:52:19.058683 31028 fs_manager.cc:359] Metadata directory not provided
I20260806 02:52:19.058807 31028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:52:19.059105 31028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:52:19.062326 31028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/master-0/data/instance:
uuid: "7e7becb94c974c04a58d14347b23e482"
format_stamp: "Formatted at 2026-08-06 02:52:19 on dist-test-slave-zj8q"
server_key: "cd76fbcd2f361fa8735bb26c2b5383b7"
server_key_iv: "a6e2af0a485d8b47d5f98380b8ee2775"
server_key_version: "encryptionkey@0"
I20260806 02:52:19.063241 31028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/master-0/wal/instance:
uuid: "7e7becb94c974c04a58d14347b23e482"
format_stamp: "Formatted at 2026-08-06 02:52:19 on dist-test-slave-zj8q"
server_key: "cd76fbcd2f361fa8735bb26c2b5383b7"
server_key_iv: "a6e2af0a485d8b47d5f98380b8ee2775"
server_key_version: "encryptionkey@0"
I20260806 02:52:19.068881 31028 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 02:52:19.072551 31047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:19.074070 31028 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260806 02:52:19.074291 31028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/master-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/master-0/wal
uuid: "7e7becb94c974c04a58d14347b23e482"
format_stamp: "Formatted at 2026-08-06 02:52:19 on dist-test-slave-zj8q"
server_key: "cd76fbcd2f361fa8735bb26c2b5383b7"
server_key_iv: "a6e2af0a485d8b47d5f98380b8ee2775"
server_key_version: "encryptionkey@0"
I20260806 02:52:19.074499 31028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:52:19.089444 31028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:52:19.093797 31028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:52:19.094091 31028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:52:19.112030 31028 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.190:37009
I20260806 02:52:19.112051 31119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.190:37009 every 8 connection(s)
I20260806 02:52:19.113492 31028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
I20260806 02:52:19.116304 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 31028
I20260806 02:52:19.116492 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/master-0/wal/instance
I20260806 02:52:19.117108 26446 external_mini_cluster.cc:1428] Setting key e75cd1e7051c3582597198460179a99d
I20260806 02:52:19.117516 31120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:19.132488 31120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e7becb94c974c04a58d14347b23e482: Bootstrap starting.
Aug 06 02:52:19 dist-test-slave-zj8q krb5kdc[31013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:52:19.136621 31120 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e7becb94c974c04a58d14347b23e482: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:19.138093 31120 log.cc:824] T 00000000000000000000000000000000 P 7e7becb94c974c04a58d14347b23e482: Log is configured to *not* fsync() on all Append() calls
I20260806 02:52:19.142185 31120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e7becb94c974c04a58d14347b23e482: No bootstrap required, opened a new log
I20260806 02:52:19.143633 31124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:19.119086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44926 (local address 127.25.211.190:37009)
0806 02:52:19.119929 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:19.119946 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:19.119998 (+ 52us) server_negotiation.cc:407] Connection header received
0806 02:52:19.122530 (+ 2532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:19.122567 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:19.123182 (+ 615us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:19.123677 (+ 495us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:19.125093 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:19.127144 (+ 2051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:19.128619 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:19.129759 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:19.129960 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:19.134608 (+ 4648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:19.134622 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:19.134644 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:19.134725 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:19.138612 (+ 3887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:19.139688 (+ 1076us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:19.139697 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:19.139704 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:19.139946 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:19.140362 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:19.140367 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:19.140370 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:19.141121 (+ 751us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:19.141446 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:19.141766 (+ 320us) server_negotiation.cc:299] Negotiation successful
0806 02:52:19.142269 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":340,"threads_started":1}
I20260806 02:52:19.150182 31120 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e7becb94c974c04a58d14347b23e482 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e7becb94c974c04a58d14347b23e482" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 37009 } }
I20260806 02:52:19.150604 31120 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e7becb94c974c04a58d14347b23e482 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:19.150681 31120 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e7becb94c974c04a58d14347b23e482 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e7becb94c974c04a58d14347b23e482, State: Initialized, Role: FOLLOWER
I20260806 02:52:19.151207 31120 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e7becb94c974c04a58d14347b23e482 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e7becb94c974c04a58d14347b23e482" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 37009 } }
I20260806 02:52:19.151432 31120 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e7becb94c974c04a58d14347b23e482 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:52:19.151531 31120 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e7becb94c974c04a58d14347b23e482 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:52:19.151662 31120 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e7becb94c974c04a58d14347b23e482 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:19.153980 31120 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e7becb94c974c04a58d14347b23e482 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e7becb94c974c04a58d14347b23e482" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 37009 } }
I20260806 02:52:19.154572 31120 leader_election.cc:302] T 00000000000000000000000000000000 P 7e7becb94c974c04a58d14347b23e482 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e7becb94c974c04a58d14347b23e482; no voters:
I20260806 02:52:19.155153 31120 leader_election.cc:288] T 00000000000000000000000000000000 P 7e7becb94c974c04a58d14347b23e482 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:52:19.155300 31127 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e7becb94c974c04a58d14347b23e482 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:52:19.155964 31127 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e7becb94c974c04a58d14347b23e482 [term 1 LEADER]: Becoming Leader. State: Replica: 7e7becb94c974c04a58d14347b23e482, State: Running, Role: LEADER
I20260806 02:52:19.156550 31127 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e7becb94c974c04a58d14347b23e482 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e7becb94c974c04a58d14347b23e482" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 37009 } }
I20260806 02:52:19.157274 31120 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e7becb94c974c04a58d14347b23e482 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:52:19.159801 31130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e7becb94c974c04a58d14347b23e482 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e7becb94c974c04a58d14347b23e482. Latest consensus state: current_term: 1 leader_uuid: "7e7becb94c974c04a58d14347b23e482" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e7becb94c974c04a58d14347b23e482" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 37009 } } }
I20260806 02:52:19.159868 31129 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e7becb94c974c04a58d14347b23e482 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e7becb94c974c04a58d14347b23e482" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e7becb94c974c04a58d14347b23e482" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 37009 } } }
I20260806 02:52:19.160218 31130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e7becb94c974c04a58d14347b23e482 [sys.catalog]: This master's current role is: LEADER
I20260806 02:52:19.160385 31129 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e7becb94c974c04a58d14347b23e482 [sys.catalog]: This master's current role is: LEADER
I20260806 02:52:19.162554 31135 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:52:19.169517 31135 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:52:19.178978 31135 catalog_manager.cc:1347] Generated new cluster ID: aac2b8b5d2114f559802eb06fbd83c3d
I20260806 02:52:19.179148 31135 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:52:19.198982 31135 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:52:19.200420 31135 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:52:19.213035 31135 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e7becb94c974c04a58d14347b23e482: Generated new TSK 0
I20260806 02:52:19.214195 31135 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.211.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab.
Entry for principal kudu/127.25.211.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab.
I20260806 02:52:19.286629 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.129:0
--local_ip_for_outbound_sockets=127.25.211.129
--webserver_interface=127.25.211.129
--webserver_port=0
--tserver_master_addrs=127.25.211.190:37009
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab
--principal=kudu/127.25.211.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:46693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:52:19.639539 31157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:52:19.639865 31157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:52:19.639993 31157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:52:19.647807 31157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:52:19.647948 31157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:52:19.648102 31157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.129
I20260806 02:52:19.658231 31157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:46693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab
--principal=kudu/127.25.211.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.211.129
--webserver_port=0
--tserver_master_addrs=127.25.211.190:37009
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.129
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:52:19.659624 31157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:52:19.660948 31157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:52:19.674609 31164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:19.674546 31167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:19.674583 31170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:52:19.675504 31157 server_base.cc:1034] running on GCE node
I20260806 02:52:19.676148 31157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:52:19.677107 31157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:52:19.678310 31157 hybrid_clock.cc:648] HybridClock initialized: now 1785984739678257 us; error 38 us; skew 500 ppm
I20260806 02:52:19.678709 31157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:52:19 dist-test-slave-zj8q krb5kdc[31013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984739, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:52:19.681986 31157 init.cc:377] Logged in from keytab as kudu/127.25.211.129@KRBTEST.COM (short username kudu)
I20260806 02:52:19.683706 31157 webserver.cc:460] Webserver started at http://127.25.211.129:36873/ using document root <none> and password file <none>
I20260806 02:52:19.684705 31157 fs_manager.cc:359] Metadata directory not provided
I20260806 02:52:19.684871 31157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:52:19.685289 31157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:52:19.688930 31157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-0/data/instance:
uuid: "238488db44784ff99235eda5220171b1"
format_stamp: "Formatted at 2026-08-06 02:52:19 on dist-test-slave-zj8q"
server_key: "515380528a34f60a34ce3c4cc320928a"
server_key_iv: "dd6fb8d091b5882e5588a2902f2dfa5a"
server_key_version: "encryptionkey@0"
I20260806 02:52:19.689772 31157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-0/wal/instance:
uuid: "238488db44784ff99235eda5220171b1"
format_stamp: "Formatted at 2026-08-06 02:52:19 on dist-test-slave-zj8q"
server_key: "515380528a34f60a34ce3c4cc320928a"
server_key_iv: "dd6fb8d091b5882e5588a2902f2dfa5a"
server_key_version: "encryptionkey@0"
I20260806 02:52:19.695371 31157 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260806 02:52:19.699198 31177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:19.700800 31157 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 02:52:19.700970 31157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-0/wal
uuid: "238488db44784ff99235eda5220171b1"
format_stamp: "Formatted at 2026-08-06 02:52:19 on dist-test-slave-zj8q"
server_key: "515380528a34f60a34ce3c4cc320928a"
server_key_iv: "dd6fb8d091b5882e5588a2902f2dfa5a"
server_key_version: "encryptionkey@0"
I20260806 02:52:19.701164 31157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:52:19.712886 31157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:52:19.717764 31157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:52:19.718065 31157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:52:19.719162 31157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:52:19.720603 31157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:52:19.720679 31157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:19.720800 31157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:52:19.720850 31157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:19.753005 31157 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.129:42791
I20260806 02:52:19.753080 31328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.129:42791 every 8 connection(s)
I20260806 02:52:19.754397 31157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
I20260806 02:52:19.763906 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 31157
I20260806 02:52:19.764139 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-0/wal/instance
I20260806 02:52:19.764853 26446 external_mini_cluster.cc:1428] Setting key 7b79aa78a01edc201ee41666e90ab8a0
Aug 06 02:52:19 dist-test-slave-zj8q krb5kdc[31013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984739, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.129@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:52:19.776639 31333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:19.757051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:37445 (local address 127.25.211.190:37009)
0806 02:52:19.757536 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:19.757587 (+ 51us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:19.758783 (+ 1196us) server_negotiation.cc:407] Connection header received
0806 02:52:19.760615 (+ 1832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:19.760619 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:19.760697 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:19.760802 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:19.762380 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:19.763143 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:19.765298 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:19.765842 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:19.765955 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:19.770623 (+ 4668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:19.770629 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:19.770633 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:19.770671 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:19.773265 (+ 2594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:19.774107 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:19.774112 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:19.774114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:19.774287 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:19.774762 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:19.774766 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:19.774768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:19.775064 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:19.775205 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:19.776020 (+ 815us) server_negotiation.cc:299] Negotiation successful
0806 02:52:19.776273 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":224,"threads_started":1}
I20260806 02:52:19.777763 31336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:19.757744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:37009 (local address 127.25.211.129:37445)
0806 02:52:19.758567 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:19.758673 (+ 106us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:19.760317 (+ 1644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:19.761008 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:19.761016 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:19.761536 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:19.762186 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:19.762214 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:19.763333 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:19.763338 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:19.765101 (+ 1763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:19.765120 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:19.766130 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:19.766138 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:19.766434 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:19.767301 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:52:19.767335 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:19.770318 (+ 2983us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:19.773445 (+ 3127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:19.773457 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:19.773474 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:19.773961 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:19.774403 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:19.774408 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:19.774412 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:19.774629 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:19.775211 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:19.775220 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:19.775613 (+ 393us) client_negotiation.cc:769] Sending connection context
0806 02:52:19.775874 (+ 261us) client_negotiation.cc:240] Negotiation successful
0806 02:52:19.776485 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":352,"threads_started":1}
I20260806 02:52:19.779510 31329 heartbeater.cc:347] Connected to a master server at 127.25.211.190:37009
I20260806 02:52:19.779848 31329 heartbeater.cc:464] Registering TS with master...
I20260806 02:52:19.780577 31329 heartbeater.cc:511] Master 127.25.211.190:37009 requested a full tablet report, sending...
I20260806 02:52:19.782847 31071 ts_manager.cc:194] Registered new tserver with Master: 238488db44784ff99235eda5220171b1 (127.25.211.129:42791)
I20260806 02:52:19.784364 31071 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.211.129@KRBTEST.COM'} at 127.25.211.129:37445
WARNING: no policy specified for kudu/127.25.211.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab.
Entry for principal kudu/127.25.211.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab.
I20260806 02:52:19.818970 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.130:0
--local_ip_for_outbound_sockets=127.25.211.130
--webserver_interface=127.25.211.130
--webserver_port=0
--tserver_master_addrs=127.25.211.190:37009
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab
--principal=kudu/127.25.211.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:46693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:52:20.167574 31340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:52:20.167891 31340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:52:20.167991 31340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:52:20.175359 31340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:52:20.175482 31340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:52:20.175595 31340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.130
I20260806 02:52:20.185732 31340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:46693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab
--principal=kudu/127.25.211.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.211.130
--webserver_port=0
--tserver_master_addrs=127.25.211.190:37009
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.130
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:52:20.187111 31340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:52:20.188436 31340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:52:20.201731 31350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:20.201797 31357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:20.201848 31353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:52:20.202571 31340 server_base.cc:1034] running on GCE node
I20260806 02:52:20.203173 31340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:52:20.204126 31340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:52:20.205335 31340 hybrid_clock.cc:648] HybridClock initialized: now 1785984740205269 us; error 55 us; skew 500 ppm
I20260806 02:52:20.205685 31340 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:52:20 dist-test-slave-zj8q krb5kdc[31013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984740, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:52:20.208967 31340 init.cc:377] Logged in from keytab as kudu/127.25.211.130@KRBTEST.COM (short username kudu)
I20260806 02:52:20.210567 31340 webserver.cc:460] Webserver started at http://127.25.211.130:33903/ using document root <none> and password file <none>
I20260806 02:52:20.211283 31340 fs_manager.cc:359] Metadata directory not provided
I20260806 02:52:20.211407 31340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:52:20.211709 31340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:52:20.214887 31340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-1/data/instance:
uuid: "e712be79458c43c298415d4eb087721c"
format_stamp: "Formatted at 2026-08-06 02:52:20 on dist-test-slave-zj8q"
server_key: "96dadfa7ba2f868b4002455501f257ec"
server_key_iv: "6c8b3a7fde0bc7fa1f88f55ad787e4c9"
server_key_version: "encryptionkey@0"
I20260806 02:52:20.215662 31340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-1/wal/instance:
uuid: "e712be79458c43c298415d4eb087721c"
format_stamp: "Formatted at 2026-08-06 02:52:20 on dist-test-slave-zj8q"
server_key: "96dadfa7ba2f868b4002455501f257ec"
server_key_iv: "6c8b3a7fde0bc7fa1f88f55ad787e4c9"
server_key_version: "encryptionkey@0"
I20260806 02:52:20.220891 31340 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 02:52:20.224548 31364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:20.225967 31340 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:52:20.226130 31340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-1/wal
uuid: "e712be79458c43c298415d4eb087721c"
format_stamp: "Formatted at 2026-08-06 02:52:20 on dist-test-slave-zj8q"
server_key: "96dadfa7ba2f868b4002455501f257ec"
server_key_iv: "6c8b3a7fde0bc7fa1f88f55ad787e4c9"
server_key_version: "encryptionkey@0"
I20260806 02:52:20.226373 31340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:52:20.241519 31340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:52:20.245878 31340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:52:20.246189 31340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:52:20.247208 31340 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:52:20.248574 31340 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:52:20.248654 31340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:20.248790 31340 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:52:20.248849 31340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:20.288646 31340 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.130:33261
I20260806 02:52:20.288683 31534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.130:33261 every 8 connection(s)
I20260806 02:52:20.290048 31340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
I20260806 02:52:20.293690 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 31340
I20260806 02:52:20.293898 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-1/wal/instance
I20260806 02:52:20.294499 26446 external_mini_cluster.cc:1428] Setting key bcf0f58d9005aca16a286f7f2bd87dc6
Aug 06 02:52:20 dist-test-slave-zj8q krb5kdc[31013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984740, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.130@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:52:20.312616 31538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:20.292770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:46367 (local address 127.25.211.190:37009)
0806 02:52:20.293252 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:20.293257 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:20.294371 (+ 1114us) server_negotiation.cc:407] Connection header received
0806 02:52:20.296770 (+ 2399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:20.296776 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:20.296861 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:20.296982 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:20.298787 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:20.299762 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:20.301385 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:20.301970 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:20.302094 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:20.306255 (+ 4161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:20.306261 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:20.306264 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:20.306303 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:20.308952 (+ 2649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:20.309986 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:20.309990 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:20.309992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:20.310206 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:20.310712 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:20.310716 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:20.310720 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:20.311013 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:20.311159 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:20.311963 (+ 804us) server_negotiation.cc:299] Negotiation successful
0806 02:52:20.312232 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":248,"threads_started":1}
I20260806 02:52:20.313592 31541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:20.293250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:37009 (local address 127.25.211.130:46367)
0806 02:52:20.294212 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:20.294255 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:20.296413 (+ 2158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:20.297281 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:20.297291 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:20.297874 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:20.298579 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:20.298601 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:20.299916 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:20.299920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:20.301228 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:20.301241 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:20.302243 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:20.302250 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:20.302472 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:20.303153 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:52:20.303178 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:20.305921 (+ 2743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:20.309148 (+ 3227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:20.309159 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:20.309175 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:20.309828 (+ 653us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:20.310347 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:20.310352 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:20.310356 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:20.310584 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:20.311150 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:20.311159 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:20.311555 (+ 396us) client_negotiation.cc:769] Sending connection context
0806 02:52:20.311828 (+ 273us) client_negotiation.cc:240] Negotiation successful
0806 02:52:20.312420 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":379,"threads_started":1}
I20260806 02:52:20.315191 31535 heartbeater.cc:347] Connected to a master server at 127.25.211.190:37009
I20260806 02:52:20.315565 31535 heartbeater.cc:464] Registering TS with master...
I20260806 02:52:20.316789 31535 heartbeater.cc:511] Master 127.25.211.190:37009 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.25.211.131@KRBTEST.COM; defaulting to no policy
I20260806 02:52:20.318668 31071 ts_manager.cc:194] Registered new tserver with Master: e712be79458c43c298415d4eb087721c (127.25.211.130:33261)
I20260806 02:52:20.319552 31071 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.211.130@KRBTEST.COM'} at 127.25.211.130:46367
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab.
Entry for principal kudu/127.25.211.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab.
I20260806 02:52:20.349860 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.131:0
--local_ip_for_outbound_sockets=127.25.211.131
--webserver_interface=127.25.211.131
--webserver_port=0
--tserver_master_addrs=127.25.211.190:37009
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab
--principal=kudu/127.25.211.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:46693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:52:20.690048 31544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:52:20.690428 31544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:52:20.690541 31544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:52:20.698050 31544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:52:20.698225 31544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:52:20.698361 31544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.131
I20260806 02:52:20.708689 31544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:46693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab
--principal=kudu/127.25.211.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.211.131
--webserver_port=0
--tserver_master_addrs=127.25.211.190:37009
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.131
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:52:20.710005 31544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:52:20.711382 31544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:52:20.724252 31551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:20.724280 31553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:20.724445 31556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:52:20.725208 31544 server_base.cc:1034] running on GCE node
I20260806 02:52:20.725900 31544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:52:20.726935 31544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:52:20.728222 31544 hybrid_clock.cc:648] HybridClock initialized: now 1785984740728173 us; error 31 us; skew 500 ppm
I20260806 02:52:20.728591 31544 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:52:20 dist-test-slave-zj8q krb5kdc[31013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984740, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:52:20.731979 31544 init.cc:377] Logged in from keytab as kudu/127.25.211.131@KRBTEST.COM (short username kudu)
I20260806 02:52:20.733549 31544 webserver.cc:460] Webserver started at http://127.25.211.131:41249/ using document root <none> and password file <none>
I20260806 02:52:20.734284 31544 fs_manager.cc:359] Metadata directory not provided
I20260806 02:52:20.734398 31544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:52:20.734675 31544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:52:20.737581 31544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-2/data/instance:
uuid: "480643b77dd1404b845c14b2fa1f1b62"
format_stamp: "Formatted at 2026-08-06 02:52:20 on dist-test-slave-zj8q"
server_key: "ddf37ce51a4fb4c4e74ad27f48fee48e"
server_key_iv: "72dfd0110a2ceab399ce2e02a4ac3086"
server_key_version: "encryptionkey@0"
I20260806 02:52:20.738432 31544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-2/wal/instance:
uuid: "480643b77dd1404b845c14b2fa1f1b62"
format_stamp: "Formatted at 2026-08-06 02:52:20 on dist-test-slave-zj8q"
server_key: "ddf37ce51a4fb4c4e74ad27f48fee48e"
server_key_iv: "72dfd0110a2ceab399ce2e02a4ac3086"
server_key_version: "encryptionkey@0"
I20260806 02:52:20.743676 31544 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260806 02:52:20.747372 31564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:20.748792 31544 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 02:52:20.748955 31544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-2/wal
uuid: "480643b77dd1404b845c14b2fa1f1b62"
format_stamp: "Formatted at 2026-08-06 02:52:20 on dist-test-slave-zj8q"
server_key: "ddf37ce51a4fb4c4e74ad27f48fee48e"
server_key_iv: "72dfd0110a2ceab399ce2e02a4ac3086"
server_key_version: "encryptionkey@0"
I20260806 02:52:20.749161 31544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:52:20.774343 31544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:52:20.778935 31544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:52:20.779237 31544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:52:20.780371 31544 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:52:20.781791 31544 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:52:20.781872 31544 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:20.781991 31544 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:52:20.782042 31544 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:20.787591 31329 heartbeater.cc:503] Master 127.25.211.190:37009 was elected leader, sending a full tablet report...
I20260806 02:52:20.816084 31544 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.131:33791
I20260806 02:52:20.816138 31727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.131:33791 every 8 connection(s)
I20260806 02:52:20.817457 31544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
I20260806 02:52:20.823665 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 31544
I20260806 02:52:20.823889 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1785984707115702-26446-0/minicluster-data/ts-2/wal/instance
I20260806 02:52:20.824483 26446 external_mini_cluster.cc:1428] Setting key f7d956cf30659eeecd60f85562d4cea4
Aug 06 02:52:20 dist-test-slave-zj8q krb5kdc[31013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984740, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.131@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:52:20.839036 31730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:20.820161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:56879 (local address 127.25.211.190:37009)
0806 02:52:20.820649 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:20.820656 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:20.821698 (+ 1042us) server_negotiation.cc:407] Connection header received
0806 02:52:20.823749 (+ 2051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:20.823764 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:20.823878 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:20.824018 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:20.825817 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:20.826877 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:20.828661 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:20.829105 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:20.829213 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:20.833198 (+ 3985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:20.833204 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:20.833207 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:20.833246 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:20.835898 (+ 2652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:20.836707 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:20.836712 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:20.836714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:20.836855 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:20.837227 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:20.837230 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:20.837232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:20.837501 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:20.837622 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:20.838404 (+ 782us) server_negotiation.cc:299] Negotiation successful
0806 02:52:20.838671 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":221,"threads_started":1}
I20260806 02:52:20.840072 31731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:20.820691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:37009 (local address 127.25.211.131:56879)
0806 02:52:20.821464 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:20.821512 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:20.823427 (+ 1915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:20.824256 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:20.824266 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:20.824883 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:20.825644 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:20.825665 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:20.827040 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:20.827044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:20.828495 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:20.828509 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:20.829347 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:20.829352 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:20.829569 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:20.830315 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:52:20.830374 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:20.832927 (+ 2553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:20.836072 (+ 3145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:20.836080 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:20.836095 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:20.836571 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:20.836947 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:20.836950 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:20.836952 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:20.837146 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:20.837596 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:20.837602 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:20.837967 (+ 365us) client_negotiation.cc:769] Sending connection context
0806 02:52:20.838264 (+ 297us) client_negotiation.cc:240] Negotiation successful
0806 02:52:20.838759 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":298,"threads_started":1}
I20260806 02:52:20.841691 31728 heartbeater.cc:347] Connected to a master server at 127.25.211.190:37009
I20260806 02:52:20.842020 31728 heartbeater.cc:464] Registering TS with master...
I20260806 02:52:20.842825 31728 heartbeater.cc:511] Master 127.25.211.190:37009 requested a full tablet report, sending...
I20260806 02:52:20.844285 31071 ts_manager.cc:194] Registered new tserver with Master: 480643b77dd1404b845c14b2fa1f1b62 (127.25.211.131:33791)
I20260806 02:52:20.845069 31071 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.211.131@KRBTEST.COM'} at 127.25.211.131:56879
I20260806 02:52:20.853200 26446 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 02:52:20 dist-test-slave-zj8q krb5kdc[31013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984740, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 06 02:52:20 dist-test-slave-zj8q krb5kdc[31013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984740, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:52:20.908232 31730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:20.894268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44932 (local address 127.25.211.190:37009)
0806 02:52:20.894510 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:20.894518 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:20.894797 (+ 279us) server_negotiation.cc:407] Connection header received
0806 02:52:20.894953 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:20.894957 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:20.895044 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:20.895149 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:20.896618 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:20.897392 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:20.898775 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:20.899187 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:20.899300 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:20.903120 (+ 3820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:20.903126 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:20.903129 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:20.903166 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:20.905752 (+ 2586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:20.906505 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:20.906509 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:20.906511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:20.906636 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:20.906962 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:20.906965 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:20.906967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:20.907299 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:20.907485 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:20.907750 (+ 265us) server_negotiation.cc:299] Negotiation successful
0806 02:52:20.907925 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260806 02:52:20.908829 31071 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:44932
I20260806 02:52:20.911098 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 31157
Aug 06 02:52:20 dist-test-slave-zj8q krb5kdc[31013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984740, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.25.211.129@KRBTEST.COM
I20260806 02:52:20.927585 31742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:20.912117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40012 (local address 127.25.211.129:42791)
0806 02:52:20.912710 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:20.912719 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:20.912745 (+ 26us) server_negotiation.cc:407] Connection header received
0806 02:52:20.912803 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:20.912810 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:20.913017 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:20.913145 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:20.914392 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:20.915422 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:20.916661 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:20.917226 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:20.917319 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:20.921541 (+ 4222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:20.921565 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:20.921582 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:20.921628 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:20.924590 (+ 2962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:20.925393 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:20.925403 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:20.925406 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:20.925609 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:20.926078 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:20.926083 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:20.926085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:20.926541 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:20.926809 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:20.927054 (+ 245us) server_negotiation.cc:299] Negotiation successful
0806 02:52:20.927279 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":288,"threads_started":1}
W20260806 02:52:20.928928 31321 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:40012
W20260806 02:52:20.930068 26446 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 31157: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 02:52:20.930310 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 31157
I20260806 02:52:20.955052 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 31340
Aug 06 02:52:20 dist-test-slave-zj8q krb5kdc[31013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984740, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.25.211.130@KRBTEST.COM
I20260806 02:52:20.971133 31743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:20.955934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53826 (local address 127.25.211.130:33261)
0806 02:52:20.956547 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:20.956554 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:20.956582 (+ 28us) server_negotiation.cc:407] Connection header received
0806 02:52:20.956645 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:20.956652 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:20.956881 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:20.957062 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:20.958395 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:20.959506 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:20.960761 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:20.961327 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:20.961427 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:20.965726 (+ 4299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:20.965740 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:20.965757 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:20.965795 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:20.968512 (+ 2717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:20.969177 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:20.969185 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:20.969188 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:20.969322 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:20.969621 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:20.969624 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:20.969626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:20.969974 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:20.970358 (+ 384us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:20.970552 (+ 194us) server_negotiation.cc:299] Negotiation successful
0806 02:52:20.970768 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":330,"threads_started":1}
W20260806 02:52:20.972648 31529 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:53826
W20260806 02:52:20.974023 26446 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 31340: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 02:52:20.974226 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 31340
I20260806 02:52:20.995558 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 31544
Aug 06 02:52:21 dist-test-slave-zj8q krb5kdc[31013](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984740, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.25.211.131@KRBTEST.COM
I20260806 02:52:21.012794 31744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:20.996686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54308 (local address 127.25.211.131:33791)
0806 02:52:20.997375 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:20.997382 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:20.997412 (+ 30us) server_negotiation.cc:407] Connection header received
0806 02:52:20.997473 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:20.997480 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:20.997690 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:20.997822 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:20.999312 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:21.000377 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:21.001741 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:21.002496 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:21.002617 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:21.007184 (+ 4567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:21.007205 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:21.007229 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:21.007279 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:21.010188 (+ 2909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:21.010903 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:21.010910 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:21.010913 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:21.011047 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:21.011332 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:21.011334 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:21.011336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:21.011689 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:21.011928 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:21.012167 (+ 239us) server_negotiation.cc:299] Negotiation successful
0806 02:52:21.012387 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":349,"threads_started":1}
W20260806 02:52:21.014035 31722 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:54308
W20260806 02:52:21.015158 26446 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 31544: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 02:52:21.015332 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 31544
I20260806 02:52:21.039667 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 31028
I20260806 02:52:21.040916 31115 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:52:21.145241 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 31028
2026-08-06T02:52:21Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3920 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:52:21 dist-test-slave-zj8q krb5kdc[31749](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:52:21 dist-test-slave-zj8q krb5kdc[31749](info): set up 2 sockets
Aug 06 02:52:21 dist-test-slave-zj8q krb5kdc[31749](info): commencing operation
krb5kdc: starting...
W20260806 02:52:22.356323 26446 mini_kdc.cc:121] Time spent starting KDC: real 1.161s user 0.006s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Aug 06 02:52:22 dist-test-slave-zj8q krb5kdc[31749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:52:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:52:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.211.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab.
Entry for principal kudu/127.25.211.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab.
I20260806 02:52:22.598733 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:45737
--webserver_interface=127.25.211.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:45585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.211.190:45737
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:52:22.946637 31765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:52:22.946973 31765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:52:22.947105 31765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:52:22.954622 31765 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:52:22.954756 31765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:52:22.954804 31765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:52:22.954890 31765 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:52:22.954947 31765 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:52:22.965399 31765 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:45585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.211.190:45737
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:45737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.211.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:52:22.966871 31765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:52:22.968223 31765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:52:22.980897 31770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:22.981045 31773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:22.980840 31771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:52:22.981941 31765 server_base.cc:1034] running on GCE node
I20260806 02:52:22.982856 31765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:52:22.984397 31765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:52:22.985606 31765 hybrid_clock.cc:648] HybridClock initialized: now 1785984742985555 us; error 45 us; skew 500 ppm
I20260806 02:52:22.985971 31765 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:52:22 dist-test-slave-zj8q krb5kdc[31749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984742, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:52:22.995301 31765 init.cc:377] Logged in from keytab as kudu/127.25.211.190@KRBTEST.COM (short username kudu)
I20260806 02:52:22.997026 31765 webserver.cc:460] Webserver started at http://127.25.211.190:35199/ using document root <none> and password file <none>
I20260806 02:52:22.997771 31765 fs_manager.cc:359] Metadata directory not provided
I20260806 02:52:22.997901 31765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:52:22.998296 31765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:52:23.001516 31765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/master-0/data/instance:
uuid: "29e0ec3241bc4f23b0657e1bdf564b66"
format_stamp: "Formatted at 2026-08-06 02:52:22 on dist-test-slave-zj8q"
server_key: "0b1a666128d787d05f27d894a9c5e144"
server_key_iv: "c623719efe197e9fa2a8d38abc7c2124"
server_key_version: "encryptionkey@0"
I20260806 02:52:23.002417 31765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/master-0/wal/instance:
uuid: "29e0ec3241bc4f23b0657e1bdf564b66"
format_stamp: "Formatted at 2026-08-06 02:52:22 on dist-test-slave-zj8q"
server_key: "0b1a666128d787d05f27d894a9c5e144"
server_key_iv: "c623719efe197e9fa2a8d38abc7c2124"
server_key_version: "encryptionkey@0"
I20260806 02:52:23.007874 31765 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 02:52:23.011811 31782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:23.013453 31765 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:52:23.013706 31765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/master-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/master-0/wal
uuid: "29e0ec3241bc4f23b0657e1bdf564b66"
format_stamp: "Formatted at 2026-08-06 02:52:22 on dist-test-slave-zj8q"
server_key: "0b1a666128d787d05f27d894a9c5e144"
server_key_iv: "c623719efe197e9fa2a8d38abc7c2124"
server_key_version: "encryptionkey@0"
I20260806 02:52:23.013922 31765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:52:23.037838 31765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:52:23.043248 31765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:52:23.043569 31765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:52:23.064375 31765 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.190:45737
I20260806 02:52:23.064412 31864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.190:45737 every 8 connection(s)
I20260806 02:52:23.065912 31765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
I20260806 02:52:23.070040 31866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:23.073585 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 31765
I20260806 02:52:23.073860 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/master-0/wal/instance
I20260806 02:52:23.074729 26446 external_mini_cluster.cc:1428] Setting key 21304c4b02fdadfa750df2be83efcb6e
I20260806 02:52:23.083447 31866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29e0ec3241bc4f23b0657e1bdf564b66: Bootstrap starting.
I20260806 02:52:23.086769 31866 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29e0ec3241bc4f23b0657e1bdf564b66: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:23.087888 31866 log.cc:824] T 00000000000000000000000000000000 P 29e0ec3241bc4f23b0657e1bdf564b66: Log is configured to *not* fsync() on all Append() calls
Aug 06 02:52:23 dist-test-slave-zj8q krb5kdc[31749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:52:23.091215 31866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29e0ec3241bc4f23b0657e1bdf564b66: No bootstrap required, opened a new log
I20260806 02:52:23.098683 31869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:23.077207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45472 (local address 127.25.211.190:45737)
0806 02:52:23.078041 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:23.078057 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:23.078104 (+ 47us) server_negotiation.cc:407] Connection header received
0806 02:52:23.079899 (+ 1795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:23.079925 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:23.080381 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:23.080794 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:23.082452 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:23.083831 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:23.085146 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:23.085866 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:23.086038 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:23.090366 (+ 4328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:23.090378 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:23.090399 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:23.090464 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:23.093823 (+ 3359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:23.094643 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:23.094651 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:23.094661 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:23.094862 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:23.095253 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:23.095257 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:23.095260 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:23.096041 (+ 781us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:23.096512 (+ 471us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:23.097029 (+ 517us) server_negotiation.cc:299] Negotiation successful
0806 02:52:23.097494 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":364,"threads_started":1}
I20260806 02:52:23.099712 31866 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29e0ec3241bc4f23b0657e1bdf564b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29e0ec3241bc4f23b0657e1bdf564b66" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45737 } }
I20260806 02:52:23.100203 31866 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29e0ec3241bc4f23b0657e1bdf564b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:23.100308 31866 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29e0ec3241bc4f23b0657e1bdf564b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29e0ec3241bc4f23b0657e1bdf564b66, State: Initialized, Role: FOLLOWER
I20260806 02:52:23.100920 31866 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29e0ec3241bc4f23b0657e1bdf564b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29e0ec3241bc4f23b0657e1bdf564b66" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45737 } }
I20260806 02:52:23.101164 31866 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29e0ec3241bc4f23b0657e1bdf564b66 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:52:23.101305 31866 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29e0ec3241bc4f23b0657e1bdf564b66 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:52:23.101465 31866 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29e0ec3241bc4f23b0657e1bdf564b66 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:23.103904 31866 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29e0ec3241bc4f23b0657e1bdf564b66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29e0ec3241bc4f23b0657e1bdf564b66" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45737 } }
I20260806 02:52:23.104491 31866 leader_election.cc:302] T 00000000000000000000000000000000 P 29e0ec3241bc4f23b0657e1bdf564b66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29e0ec3241bc4f23b0657e1bdf564b66; no voters:
I20260806 02:52:23.105144 31866 leader_election.cc:288] T 00000000000000000000000000000000 P 29e0ec3241bc4f23b0657e1bdf564b66 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:52:23.105288 31872 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29e0ec3241bc4f23b0657e1bdf564b66 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:52:23.106231 31872 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29e0ec3241bc4f23b0657e1bdf564b66 [term 1 LEADER]: Becoming Leader. State: Replica: 29e0ec3241bc4f23b0657e1bdf564b66, State: Running, Role: LEADER
I20260806 02:52:23.106786 31872 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29e0ec3241bc4f23b0657e1bdf564b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29e0ec3241bc4f23b0657e1bdf564b66" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45737 } }
I20260806 02:52:23.107512 31866 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29e0ec3241bc4f23b0657e1bdf564b66 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:52:23.110910 31875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29e0ec3241bc4f23b0657e1bdf564b66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29e0ec3241bc4f23b0657e1bdf564b66. Latest consensus state: current_term: 1 leader_uuid: "29e0ec3241bc4f23b0657e1bdf564b66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29e0ec3241bc4f23b0657e1bdf564b66" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45737 } } }
I20260806 02:52:23.111403 31875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29e0ec3241bc4f23b0657e1bdf564b66 [sys.catalog]: This master's current role is: LEADER
I20260806 02:52:23.113560 31882 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:52:23.111214 31873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29e0ec3241bc4f23b0657e1bdf564b66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29e0ec3241bc4f23b0657e1bdf564b66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29e0ec3241bc4f23b0657e1bdf564b66" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45737 } } }
I20260806 02:52:23.116149 31873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29e0ec3241bc4f23b0657e1bdf564b66 [sys.catalog]: This master's current role is: LEADER
I20260806 02:52:23.119037 31882 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:52:23.130108 31882 catalog_manager.cc:1347] Generated new cluster ID: 3fddb7a6be80484ea5055f096843f7c2
I20260806 02:52:23.130291 31882 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:52:23.155264 31882 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:52:23.156894 31882 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:52:23.163051 31882 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29e0ec3241bc4f23b0657e1bdf564b66: Generated new TSK 0
I20260806 02:52:23.163942 31882 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.211.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab.
Entry for principal kudu/127.25.211.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab.
I20260806 02:52:23.233474 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.129:0
--local_ip_for_outbound_sockets=127.25.211.129
--webserver_interface=127.25.211.129
--webserver_port=0
--tserver_master_addrs=127.25.211.190:45737
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab
--principal=kudu/127.25.211.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:45585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:52:23.601423 31900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:52:23.601775 31900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:52:23.601914 31900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:52:23.609846 31900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:52:23.610008 31900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:52:23.610201 31900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.129
I20260806 02:52:23.620862 31900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:45585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab
--principal=kudu/127.25.211.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.211.129
--webserver_port=0
--tserver_master_addrs=127.25.211.190:45737
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.129
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:52:23.622301 31900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:52:23.623625 31900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:52:23.637593 31906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:23.637615 31907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:23.637595 31909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:52:23.638854 31900 server_base.cc:1034] running on GCE node
I20260806 02:52:23.639562 31900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:52:23.640718 31900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:52:23.641969 31900 hybrid_clock.cc:648] HybridClock initialized: now 1785984743641903 us; error 59 us; skew 500 ppm
I20260806 02:52:23.642472 31900 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:52:23 dist-test-slave-zj8q krb5kdc[31749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984743, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:52:23.646453 31900 init.cc:377] Logged in from keytab as kudu/127.25.211.129@KRBTEST.COM (short username kudu)
I20260806 02:52:23.648123 31900 webserver.cc:460] Webserver started at http://127.25.211.129:34453/ using document root <none> and password file <none>
I20260806 02:52:23.648878 31900 fs_manager.cc:359] Metadata directory not provided
I20260806 02:52:23.649009 31900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:52:23.649333 31900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:52:23.652755 31900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/ts-0/data/instance:
uuid: "1bfdc6036f084642be936057585a339a"
format_stamp: "Formatted at 2026-08-06 02:52:23 on dist-test-slave-zj8q"
server_key: "da22d0c168d00166b0814ddedae1a886"
server_key_iv: "ea0bf679e14d492d4756f364f1363a43"
server_key_version: "encryptionkey@0"
I20260806 02:52:23.653661 31900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/ts-0/wal/instance:
uuid: "1bfdc6036f084642be936057585a339a"
format_stamp: "Formatted at 2026-08-06 02:52:23 on dist-test-slave-zj8q"
server_key: "da22d0c168d00166b0814ddedae1a886"
server_key_iv: "ea0bf679e14d492d4756f364f1363a43"
server_key_version: "encryptionkey@0"
I20260806 02:52:23.659915 31900 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.003s
I20260806 02:52:23.664516 31917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:23.666275 31900 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 02:52:23.666461 31900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/ts-0/wal
uuid: "1bfdc6036f084642be936057585a339a"
format_stamp: "Formatted at 2026-08-06 02:52:23 on dist-test-slave-zj8q"
server_key: "da22d0c168d00166b0814ddedae1a886"
server_key_iv: "ea0bf679e14d492d4756f364f1363a43"
server_key_version: "encryptionkey@0"
I20260806 02:52:23.666676 31900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:52:23.693601 31900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:52:23.698807 31900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:52:23.699169 31900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:52:23.700716 31900 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:52:23.702270 31900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:52:23.702355 31900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:23.702448 31900 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:52:23.702491 31900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:23.736403 31900 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.129:40967
I20260806 02:52:23.736452 32087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.129:40967 every 8 connection(s)
I20260806 02:52:23.737792 31900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
I20260806 02:52:23.742671 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 31900
I20260806 02:52:23.742899 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/ts-0/wal/instance
I20260806 02:52:23.743520 26446 external_mini_cluster.cc:1428] Setting key f008faeb42fa2b4c9aab67f4f0cb82ac
Aug 06 02:52:23 dist-test-slave-zj8q krb5kdc[31749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984743, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.129@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:52:23.759178 32094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:23.740536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:46561 (local address 127.25.211.190:45737)
0806 02:52:23.740979 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:23.740983 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:23.741985 (+ 1002us) server_negotiation.cc:407] Connection header received
0806 02:52:23.744027 (+ 2042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:23.744033 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:23.744120 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:23.744222 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:23.745952 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:23.746789 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:23.748556 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:23.748996 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:23.749120 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:23.752882 (+ 3762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:23.752888 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:23.752891 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:23.752929 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:23.755828 (+ 2899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:23.756676 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:23.756681 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:23.756683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:23.756819 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:23.757271 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:23.757274 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:23.757277 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:23.757558 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:23.757703 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:23.758540 (+ 837us) server_negotiation.cc:299] Negotiation successful
0806 02:52:23.758815 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":239,"threads_started":1}
I20260806 02:52:23.760252 32095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:23.741035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45737 (local address 127.25.211.129:46561)
0806 02:52:23.741811 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:23.741853 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:23.743734 (+ 1881us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:23.744497 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:23.744506 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:23.745083 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:23.745769 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:23.745792 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:23.746973 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:23.746977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:23.748378 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:23.748394 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:23.749267 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:23.749275 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:23.749530 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:23.750264 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:52:23.750291 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:23.752606 (+ 2315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:23.756011 (+ 3405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:23.756022 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:23.756039 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:23.756537 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:23.756933 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:23.756938 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:23.756943 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:23.757160 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:23.757696 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:23.757705 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:23.758094 (+ 389us) client_negotiation.cc:769] Sending connection context
0806 02:52:23.758423 (+ 329us) client_negotiation.cc:240] Negotiation successful
0806 02:52:23.759031 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":288,"threads_started":1}
I20260806 02:52:23.762079 32089 heartbeater.cc:347] Connected to a master server at 127.25.211.190:45737
I20260806 02:52:23.762434 32089 heartbeater.cc:464] Registering TS with master...
I20260806 02:52:23.763247 32089 heartbeater.cc:511] Master 127.25.211.190:45737 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.25.211.130@KRBTEST.COM; defaulting to no policy
I20260806 02:52:23.765569 31810 ts_manager.cc:194] Registered new tserver with Master: 1bfdc6036f084642be936057585a339a (127.25.211.129:40967)
I20260806 02:52:23.767238 31810 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.211.129@KRBTEST.COM'} at 127.25.211.129:46561
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab.
Entry for principal kudu/127.25.211.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab.
I20260806 02:52:23.797586 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.130:0
--local_ip_for_outbound_sockets=127.25.211.130
--webserver_interface=127.25.211.130
--webserver_port=0
--tserver_master_addrs=127.25.211.190:45737
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab
--principal=kudu/127.25.211.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:45585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:52:24.170439 32100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:52:24.170763 32100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:52:24.170909 32100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:52:24.179075 32100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:52:24.179198 32100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:52:24.179343 32100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.130
I20260806 02:52:24.189503 32100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:45585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab
--principal=kudu/127.25.211.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.211.130
--webserver_port=0
--tserver_master_addrs=127.25.211.190:45737
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.130
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:52:24.190922 32100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:52:24.192302 32100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:52:24.206287 32108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:24.206351 32107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:24.206328 32111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:52:24.208207 32100 server_base.cc:1034] running on GCE node
I20260806 02:52:24.208940 32100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:52:24.210011 32100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:52:24.211252 32100 hybrid_clock.cc:648] HybridClock initialized: now 1785984744211192 us; error 46 us; skew 500 ppm
I20260806 02:52:24.211587 32100 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:52:24 dist-test-slave-zj8q krb5kdc[31749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984744, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:52:24.215173 32100 init.cc:377] Logged in from keytab as kudu/127.25.211.130@KRBTEST.COM (short username kudu)
I20260806 02:52:24.216739 32100 webserver.cc:460] Webserver started at http://127.25.211.130:41037/ using document root <none> and password file <none>
I20260806 02:52:24.217531 32100 fs_manager.cc:359] Metadata directory not provided
I20260806 02:52:24.217661 32100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:52:24.217978 32100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:52:24.221372 32100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/ts-1/data/instance:
uuid: "002eebb848b8413581e9c78ada37b8e7"
format_stamp: "Formatted at 2026-08-06 02:52:24 on dist-test-slave-zj8q"
server_key: "12cc39b13eb3cf7c152888533ac73364"
server_key_iv: "fbc40f7d364f4edaf142c67dec0ad881"
server_key_version: "encryptionkey@0"
I20260806 02:52:24.222637 32100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/ts-1/wal/instance:
uuid: "002eebb848b8413581e9c78ada37b8e7"
format_stamp: "Formatted at 2026-08-06 02:52:24 on dist-test-slave-zj8q"
server_key: "12cc39b13eb3cf7c152888533ac73364"
server_key_iv: "fbc40f7d364f4edaf142c67dec0ad881"
server_key_version: "encryptionkey@0"
I20260806 02:52:24.228286 32100 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 02:52:24.231916 32121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:24.233341 32100 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 02:52:24.233505 32100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/ts-1/wal
uuid: "002eebb848b8413581e9c78ada37b8e7"
format_stamp: "Formatted at 2026-08-06 02:52:24 on dist-test-slave-zj8q"
server_key: "12cc39b13eb3cf7c152888533ac73364"
server_key_iv: "fbc40f7d364f4edaf142c67dec0ad881"
server_key_version: "encryptionkey@0"
I20260806 02:52:24.233722 32100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:52:24.255295 32100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:52:24.260221 32100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:52:24.260522 32100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:52:24.261655 32100 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:52:24.263224 32100 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:52:24.263309 32100 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:24.263415 32100 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:52:24.263465 32100 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:24.296203 32100 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.130:42589
I20260806 02:52:24.296277 32276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.130:42589 every 8 connection(s)
I20260806 02:52:24.297526 32100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
I20260806 02:52:24.306018 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 32100
I20260806 02:52:24.306270 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/ts-1/wal/instance
I20260806 02:52:24.306740 26446 external_mini_cluster.cc:1428] Setting key 38e6139b1499e5563f02a27910ed194e
Aug 06 02:52:24 dist-test-slave-zj8q krb5kdc[31749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984744, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.130@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:52:24.319407 32281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:24.300168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:48993 (local address 127.25.211.190:45737)
0806 02:52:24.300609 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:24.300614 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:24.301507 (+ 893us) server_negotiation.cc:407] Connection header received
0806 02:52:24.303421 (+ 1914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:24.303427 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:24.303514 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:24.303613 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:24.305186 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:24.306110 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:24.307979 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:24.308666 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:24.308782 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:24.312845 (+ 4063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:24.312852 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:24.312857 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:24.312901 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:24.316025 (+ 3124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:24.316854 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:24.316859 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:24.316862 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:24.317017 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:24.317495 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:24.317500 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:24.317503 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:24.317864 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:24.317981 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:24.318955 (+ 974us) server_negotiation.cc:299] Negotiation successful
0806 02:52:24.319144 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":235,"threads_started":1}
I20260806 02:52:24.320848 32282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:24.300580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45737 (local address 127.25.211.130:48993)
0806 02:52:24.301329 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:24.301370 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:24.303137 (+ 1767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:24.303866 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:24.303875 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:24.304418 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:24.304993 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:24.305016 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:24.306322 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:24.306328 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:24.307842 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:24.307856 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:24.308905 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:24.308913 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:24.309148 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:24.309887 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:52:24.309912 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:24.312539 (+ 2627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:24.316218 (+ 3679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:24.316227 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:24.316242 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:24.316723 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:24.317116 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:24.317119 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:24.317121 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:24.317360 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:24.317998 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:24.318007 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:24.318541 (+ 534us) client_negotiation.cc:769] Sending connection context
0806 02:52:24.318884 (+ 343us) client_negotiation.cc:240] Negotiation successful
0806 02:52:24.319459 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":301,"threads_started":1}
I20260806 02:52:24.322377 32277 heartbeater.cc:347] Connected to a master server at 127.25.211.190:45737
I20260806 02:52:24.322803 32277 heartbeater.cc:464] Registering TS with master...
I20260806 02:52:24.323753 32277 heartbeater.cc:511] Master 127.25.211.190:45737 requested a full tablet report, sending...
I20260806 02:52:24.325349 31810 ts_manager.cc:194] Registered new tserver with Master: 002eebb848b8413581e9c78ada37b8e7 (127.25.211.130:42589)
I20260806 02:52:24.326277 31810 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.211.130@KRBTEST.COM'} at 127.25.211.130:48993
WARNING: no policy specified for kudu/127.25.211.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab.
Entry for principal kudu/127.25.211.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab.
I20260806 02:52:24.361765 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.131:0
--local_ip_for_outbound_sockets=127.25.211.131
--webserver_interface=127.25.211.131
--webserver_port=0
--tserver_master_addrs=127.25.211.190:45737
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab
--principal=kudu/127.25.211.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:45585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:52:24.705451 32287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:52:24.705798 32287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:52:24.705951 32287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:52:24.713714 32287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:52:24.713869 32287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:52:24.714030 32287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.131
I20260806 02:52:24.724190 32287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:45585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab
--principal=kudu/127.25.211.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.211.131
--webserver_port=0
--tserver_master_addrs=127.25.211.190:45737
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.131
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:52:24.725592 32287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:52:24.727048 32287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:52:24.740842 32296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:24.740903 32297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:24.741962 32299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:52:24.742427 32287 server_base.cc:1034] running on GCE node
I20260806 02:52:24.743078 32287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:52:24.744060 32287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:52:24.745286 32287 hybrid_clock.cc:648] HybridClock initialized: now 1785984744745227 us; error 51 us; skew 500 ppm
I20260806 02:52:24.745635 32287 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:52:24 dist-test-slave-zj8q krb5kdc[31749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984744, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:52:24.749001 32287 init.cc:377] Logged in from keytab as kudu/127.25.211.131@KRBTEST.COM (short username kudu)
I20260806 02:52:24.750605 32287 webserver.cc:460] Webserver started at http://127.25.211.131:44149/ using document root <none> and password file <none>
I20260806 02:52:24.751331 32287 fs_manager.cc:359] Metadata directory not provided
I20260806 02:52:24.751454 32287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:52:24.751763 32287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:52:24.754880 32287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/ts-2/data/instance:
uuid: "072fb9ded2834a88bec566e97d11218a"
format_stamp: "Formatted at 2026-08-06 02:52:24 on dist-test-slave-zj8q"
server_key: "95e2661318522c8ed6c6e0673a1a4bd9"
server_key_iv: "52a3abb84cc64348cadaccc4d9a515ad"
server_key_version: "encryptionkey@0"
I20260806 02:52:24.755700 32287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/ts-2/wal/instance:
uuid: "072fb9ded2834a88bec566e97d11218a"
format_stamp: "Formatted at 2026-08-06 02:52:24 on dist-test-slave-zj8q"
server_key: "95e2661318522c8ed6c6e0673a1a4bd9"
server_key_iv: "52a3abb84cc64348cadaccc4d9a515ad"
server_key_version: "encryptionkey@0"
I20260806 02:52:24.761475 32287 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 02:52:24.765607 32310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:24.767244 32287 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:52:24.767443 32287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/ts-2/wal
uuid: "072fb9ded2834a88bec566e97d11218a"
format_stamp: "Formatted at 2026-08-06 02:52:24 on dist-test-slave-zj8q"
server_key: "95e2661318522c8ed6c6e0673a1a4bd9"
server_key_iv: "52a3abb84cc64348cadaccc4d9a515ad"
server_key_version: "encryptionkey@0"
I20260806 02:52:24.767668 32287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:52:24.771195 32089 heartbeater.cc:503] Master 127.25.211.190:45737 was elected leader, sending a full tablet report...
I20260806 02:52:24.780563 32287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:52:24.785027 32287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:52:24.785336 32287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:52:24.786417 32287 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:52:24.787809 32287 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:52:24.787889 32287 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:24.788018 32287 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:52:24.788079 32287 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:24.821471 32287 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.131:36343
I20260806 02:52:24.821580 32489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.131:36343 every 8 connection(s)
I20260806 02:52:24.822934 32287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
I20260806 02:52:24.826471 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 32287
I20260806 02:52:24.826666 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1785984707115702-26446-0/minicluster-data/ts-2/wal/instance
I20260806 02:52:24.827219 26446 external_mini_cluster.cc:1428] Setting key bfc84c39327806a4fcecca4d103061f3
Aug 06 02:52:24 dist-test-slave-zj8q krb5kdc[31749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984744, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.131@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:52:24.845628 32493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:24.825834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:40629 (local address 127.25.211.190:45737)
0806 02:52:24.826393 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:24.826399 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:24.827630 (+ 1231us) server_negotiation.cc:407] Connection header received
0806 02:52:24.829979 (+ 2349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:24.829983 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:24.830057 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:24.830179 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:24.832042 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:24.832878 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:24.834668 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:24.835112 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:24.835224 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:24.839047 (+ 3823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:24.839053 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:24.839056 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:24.839094 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:24.841949 (+ 2855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:24.842980 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:24.842986 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:24.842989 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:24.843167 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:24.843681 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:24.843687 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:24.843689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:24.843999 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:24.844149 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:24.844969 (+ 820us) server_negotiation.cc:299] Negotiation successful
0806 02:52:24.845234 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":258,"threads_started":1}
I20260806 02:52:24.846643 32494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:24.826490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45737 (local address 127.25.211.131:40629)
0806 02:52:24.827413 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:24.827486 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:24.829609 (+ 2123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:24.830487 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:24.830497 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:24.831185 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:24.831837 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:24.831859 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:24.833061 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:24.833065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:24.834471 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:24.834490 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:24.835368 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:24.835374 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:24.835617 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:24.836338 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:52:24.836363 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:24.838782 (+ 2419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:24.842117 (+ 3335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:24.842127 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:24.842144 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:24.842839 (+ 695us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:24.843285 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:24.843290 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:24.843294 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:24.843548 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:24.844142 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:24.844151 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:24.844545 (+ 394us) client_negotiation.cc:769] Sending connection context
0806 02:52:24.844817 (+ 272us) client_negotiation.cc:240] Negotiation successful
0806 02:52:24.845403 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":347,"threads_started":1}
I20260806 02:52:24.848079 32490 heartbeater.cc:347] Connected to a master server at 127.25.211.190:45737
I20260806 02:52:24.848410 32490 heartbeater.cc:464] Registering TS with master...
I20260806 02:52:24.849135 32490 heartbeater.cc:511] Master 127.25.211.190:45737 requested a full tablet report, sending...
I20260806 02:52:24.851034 31810 ts_manager.cc:194] Registered new tserver with Master: 072fb9ded2834a88bec566e97d11218a (127.25.211.131:36343)
I20260806 02:52:24.851890 31810 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.211.131@KRBTEST.COM'} at 127.25.211.131:40629
I20260806 02:52:24.856493 26446 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 02:52:24 dist-test-slave-zj8q krb5kdc[31749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 02:52:24 dist-test-slave-zj8q krb5kdc[31749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.129@KRBTEST.COM
I20260806 02:52:24.915931 32501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:24.899126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34344 (local address 127.25.211.129:40967)
0806 02:52:24.899790 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:24.899798 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:24.899828 (+ 30us) server_negotiation.cc:407] Connection header received
0806 02:52:24.899896 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:24.899903 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:24.900120 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:24.900280 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:24.901745 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:24.902880 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:24.904165 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:24.904794 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:24.904898 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:24.909748 (+ 4850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:24.909766 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:24.909784 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:24.909835 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:24.912987 (+ 3152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:24.913815 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:24.913823 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:24.913825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:24.913974 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:24.914480 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:24.914484 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:24.914486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:24.914877 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:24.915130 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:24.915415 (+ 285us) server_negotiation.cc:299] Negotiation successful
0806 02:52:24.915637 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":334,"threads_started":1}
Aug 06 02:52:24 dist-test-slave-zj8q krb5kdc[31749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:52:24.934705 32493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:24.920648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45474 (local address 127.25.211.190:45737)
0806 02:52:24.920854 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:24.920861 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:24.921140 (+ 279us) server_negotiation.cc:407] Connection header received
0806 02:52:24.921314 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:24.921319 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:24.921385 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:24.921469 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:24.922870 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:24.923615 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:24.924835 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:24.925253 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:24.925370 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:24.929376 (+ 4006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:24.929381 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:24.929384 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:24.929443 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:24.931961 (+ 2518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:24.932680 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:24.932684 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:24.932687 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:24.932832 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:24.933200 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:24.933204 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:24.933206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:24.933560 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:24.933814 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:24.934126 (+ 312us) server_negotiation.cc:299] Negotiation successful
0806 02:52:24.934394 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260806 02:52:24.935386 31810 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45474
I20260806 02:52:24.936480 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 31900
I20260806 02:52:24.949677 32501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:24.937417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34360 (local address 127.25.211.129:40967)
0806 02:52:24.937629 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:24.937635 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:24.937843 (+ 208us) server_negotiation.cc:407] Connection header received
0806 02:52:24.937983 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:24.937987 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:24.938096 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:24.938209 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:24.939441 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:24.940190 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:24.941508 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:24.941951 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:24.942064 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:24.944425 (+ 2361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:24.944430 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:24.944434 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:24.944470 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:24.947086 (+ 2616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:24.947757 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:24.947761 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:24.947763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:24.947893 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:24.948271 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:24.948275 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:24.948277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:24.948540 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:24.948724 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:24.949026 (+ 302us) server_negotiation.cc:299] Negotiation successful
0806 02:52:24.949232 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 02:52:24.950412 32082 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:52:25.034207 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 31900
I20260806 02:52:25.057300 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 32100
Aug 06 02:52:25 dist-test-slave-zj8q krb5kdc[31749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.130@KRBTEST.COM
I20260806 02:52:25.075887 32511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:25.058342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47940 (local address 127.25.211.130:42589)
0806 02:52:25.058996 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:25.059003 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:25.059047 (+ 44us) server_negotiation.cc:407] Connection header received
0806 02:52:25.059116 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:25.059123 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:25.059354 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:25.059491 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:25.060791 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:25.061864 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:25.063318 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:25.064169 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:25.064322 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:25.068712 (+ 4390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:25.068733 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:25.068755 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:25.068815 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:25.072840 (+ 4025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:25.073646 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:25.073655 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:25.073657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:25.073807 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:25.074251 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:25.074255 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:25.074258 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:25.074665 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:25.074910 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:25.075288 (+ 378us) server_negotiation.cc:299] Negotiation successful
0806 02:52:25.075542 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":344,"threads_started":1}
I20260806 02:52:25.077183 32270 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:52:25.160930 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 32100
I20260806 02:52:25.182680 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 32287
Aug 06 02:52:25 dist-test-slave-zj8q krb5kdc[31749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.131@KRBTEST.COM
I20260806 02:52:25.200287 32513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:25.183756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51694 (local address 127.25.211.131:36343)
0806 02:52:25.184438 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:25.184446 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:25.184477 (+ 31us) server_negotiation.cc:407] Connection header received
0806 02:52:25.184544 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:25.184551 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:25.184794 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:25.184960 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:25.186379 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:25.187479 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:25.188847 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:25.189461 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:25.189598 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:25.194304 (+ 4706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:25.194323 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:25.194341 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:25.194386 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:25.197417 (+ 3031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:25.198316 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:25.198325 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:25.198327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:25.198479 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:25.198849 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:25.198853 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:25.198855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:25.199234 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:25.199489 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:25.199743 (+ 254us) server_negotiation.cc:299] Negotiation successful
0806 02:52:25.199988 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":348,"threads_started":1}
I20260806 02:52:25.201644 32482 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:52:25.286212 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 32287
I20260806 02:52:25.309484 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 31765
I20260806 02:52:25.310745 31859 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:52:25.415642 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 31765
2026-08-06T02:52:25Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4267 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:52:25 dist-test-slave-zj8q krb5kdc[32523](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:52:25 dist-test-slave-zj8q krb5kdc[32523](info): set up 2 sockets
Aug 06 02:52:25 dist-test-slave-zj8q krb5kdc[32523](info): commencing operation
krb5kdc: starting...
W20260806 02:52:26.628978 26446 mini_kdc.cc:121] Time spent starting KDC: real 1.166s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Aug 06 02:52:26 dist-test-slave-zj8q krb5kdc[32523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:52:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:52:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.211.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab.
Entry for principal kudu/127.25.211.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab.
I20260806 02:52:26.887100 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:41447
--webserver_interface=127.25.211.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:40227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.211.190:41447
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:52:27.229454 32541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:52:27.229808 32541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:52:27.229952 32541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:52:27.237741 32541 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:52:27.237923 32541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:52:27.238016 32541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:52:27.238123 32541 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:52:27.238225 32541 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:52:27.248914 32541 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:40227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.211.190:41447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:41447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.211.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:52:27.250453 32541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:52:27.251876 32541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:52:27.264307 32547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:27.264344 32549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:27.264199 32551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:52:27.265640 32541 server_base.cc:1034] running on GCE node
I20260806 02:52:27.266674 32541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:52:27.268334 32541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:52:27.269608 32541 hybrid_clock.cc:648] HybridClock initialized: now 1785984747269530 us; error 54 us; skew 500 ppm
I20260806 02:52:27.270035 32541 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:52:27 dist-test-slave-zj8q krb5kdc[32523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984747, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:52:27.273676 32541 init.cc:377] Logged in from keytab as kudu/127.25.211.190@KRBTEST.COM (short username kudu)
I20260806 02:52:27.275053 32541 fs_manager.cc:359] Metadata directory not provided
I20260806 02:52:27.275193 32541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:52:27.275554 32541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:52:27.278965 32541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/master-0/data/instance:
uuid: "c34a097779344299be8a6b87d6a1c232"
format_stamp: "Formatted at 2026-08-06 02:52:27 on dist-test-slave-zj8q"
server_key: "0f35a8636238f1ef3bc6fa53eadc5b28"
server_key_iv: "fd23a0bcd469f4dd17ccfd8896ac40e3"
server_key_version: "encryptionkey@0"
I20260806 02:52:27.279860 32541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/master-0/wal/instance:
uuid: "c34a097779344299be8a6b87d6a1c232"
format_stamp: "Formatted at 2026-08-06 02:52:27 on dist-test-slave-zj8q"
server_key: "0f35a8636238f1ef3bc6fa53eadc5b28"
server_key_iv: "fd23a0bcd469f4dd17ccfd8896ac40e3"
server_key_version: "encryptionkey@0"
I20260806 02:52:27.285640 32541 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 02:52:27.289983 32558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:27.291647 32541 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260806 02:52:27.291855 32541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/master-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/master-0/wal
uuid: "c34a097779344299be8a6b87d6a1c232"
format_stamp: "Formatted at 2026-08-06 02:52:27 on dist-test-slave-zj8q"
server_key: "0f35a8636238f1ef3bc6fa53eadc5b28"
server_key_iv: "fd23a0bcd469f4dd17ccfd8896ac40e3"
server_key_version: "encryptionkey@0"
I20260806 02:52:27.292126 32541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:52:27.322351 32541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:52:27.326934 32541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:52:27.327229 32541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:52:27.345216 32541 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.190:41447
I20260806 02:52:27.345240 32647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.190:41447 every 8 connection(s)
I20260806 02:52:27.346226 32541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
I20260806 02:52:27.350193 32648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:27.352696 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 32541
I20260806 02:52:27.352919 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/master-0/wal/instance
I20260806 02:52:27.353526 26446 external_mini_cluster.cc:1428] Setting key 251f82494812dbc511ecd079c0f67102
I20260806 02:52:27.363049 32648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c34a097779344299be8a6b87d6a1c232: Bootstrap starting.
I20260806 02:52:27.366506 32648 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c34a097779344299be8a6b87d6a1c232: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:27.367586 32648 log.cc:824] T 00000000000000000000000000000000 P c34a097779344299be8a6b87d6a1c232: Log is configured to *not* fsync() on all Append() calls
Aug 06 02:52:27 dist-test-slave-zj8q krb5kdc[32523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:52:27.371078 32648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c34a097779344299be8a6b87d6a1c232: No bootstrap required, opened a new log
I20260806 02:52:27.378810 32652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:27.355635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43530 (local address 127.25.211.190:41447)
0806 02:52:27.356493 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:27.356505 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:27.356549 (+ 44us) server_negotiation.cc:407] Connection header received
0806 02:52:27.358293 (+ 1744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:27.358318 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:27.358757 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:27.359134 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:27.361111 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:27.362566 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:27.363989 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:27.364758 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:27.364940 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:27.369546 (+ 4606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:27.369554 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:27.369572 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:27.369611 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:27.373428 (+ 3817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:27.374343 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:27.374354 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:27.374364 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:27.374602 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:27.375032 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:27.375038 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:27.375041 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:27.375861 (+ 820us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:27.376166 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:27.376738 (+ 572us) server_negotiation.cc:299] Negotiation successful
0806 02:52:27.377254 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":399,"threads_started":1}
I20260806 02:52:27.381136 32648 raft_consensus.cc:348] T 00000000000000000000000000000000 P c34a097779344299be8a6b87d6a1c232 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c34a097779344299be8a6b87d6a1c232" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 41447 } }
I20260806 02:52:27.381526 32648 raft_consensus.cc:374] T 00000000000000000000000000000000 P c34a097779344299be8a6b87d6a1c232 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:27.381668 32648 raft_consensus.cc:729] T 00000000000000000000000000000000 P c34a097779344299be8a6b87d6a1c232 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c34a097779344299be8a6b87d6a1c232, State: Initialized, Role: FOLLOWER
I20260806 02:52:27.382267 32648 consensus_queue.cc:260] T 00000000000000000000000000000000 P c34a097779344299be8a6b87d6a1c232 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c34a097779344299be8a6b87d6a1c232" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 41447 } }
I20260806 02:52:27.382503 32648 raft_consensus.cc:388] T 00000000000000000000000000000000 P c34a097779344299be8a6b87d6a1c232 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:52:27.382632 32648 raft_consensus.cc:482] T 00000000000000000000000000000000 P c34a097779344299be8a6b87d6a1c232 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:52:27.382784 32648 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c34a097779344299be8a6b87d6a1c232 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:27.385003 32648 raft_consensus.cc:504] T 00000000000000000000000000000000 P c34a097779344299be8a6b87d6a1c232 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c34a097779344299be8a6b87d6a1c232" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 41447 } }
I20260806 02:52:27.385557 32648 leader_election.cc:302] T 00000000000000000000000000000000 P c34a097779344299be8a6b87d6a1c232 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c34a097779344299be8a6b87d6a1c232; no voters:
I20260806 02:52:27.386195 32648 leader_election.cc:288] T 00000000000000000000000000000000 P c34a097779344299be8a6b87d6a1c232 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:52:27.386394 32655 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c34a097779344299be8a6b87d6a1c232 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:52:27.387058 32655 raft_consensus.cc:686] T 00000000000000000000000000000000 P c34a097779344299be8a6b87d6a1c232 [term 1 LEADER]: Becoming Leader. State: Replica: c34a097779344299be8a6b87d6a1c232, State: Running, Role: LEADER
I20260806 02:52:27.387729 32655 consensus_queue.cc:237] T 00000000000000000000000000000000 P c34a097779344299be8a6b87d6a1c232 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c34a097779344299be8a6b87d6a1c232" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 41447 } }
I20260806 02:52:27.388422 32648 sys_catalog.cc:564] T 00000000000000000000000000000000 P c34a097779344299be8a6b87d6a1c232 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:52:27.390707 32659 sys_catalog.cc:455] T 00000000000000000000000000000000 P c34a097779344299be8a6b87d6a1c232 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c34a097779344299be8a6b87d6a1c232" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c34a097779344299be8a6b87d6a1c232" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 41447 } } }
I20260806 02:52:27.391234 32659 sys_catalog.cc:458] T 00000000000000000000000000000000 P c34a097779344299be8a6b87d6a1c232 [sys.catalog]: This master's current role is: LEADER
I20260806 02:52:27.392445 32660 sys_catalog.cc:455] T 00000000000000000000000000000000 P c34a097779344299be8a6b87d6a1c232 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c34a097779344299be8a6b87d6a1c232. Latest consensus state: current_term: 1 leader_uuid: "c34a097779344299be8a6b87d6a1c232" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c34a097779344299be8a6b87d6a1c232" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 41447 } } }
I20260806 02:52:27.392804 32660 sys_catalog.cc:458] T 00000000000000000000000000000000 P c34a097779344299be8a6b87d6a1c232 [sys.catalog]: This master's current role is: LEADER
I20260806 02:52:27.393517 32668 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:52:27.399082 32668 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:52:27.408804 32668 catalog_manager.cc:1347] Generated new cluster ID: 006f4a673cf84e00a29a3a8ca149245f
I20260806 02:52:27.408936 32668 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:52:27.421972 32668 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:52:27.423573 32668 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:52:27.431402 32668 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c34a097779344299be8a6b87d6a1c232: Generated new TSK 0
I20260806 02:52:27.432345 32668 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.211.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab.
Entry for principal kudu/127.25.211.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab.
I20260806 02:52:27.494860 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.129:0
--local_ip_for_outbound_sockets=127.25.211.129
--webserver_interface=127.25.211.129
--webserver_port=0
--tserver_master_addrs=127.25.211.190:41447
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab
--principal=kudu/127.25.211.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:40227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:52:27.853564 32688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:52:27.853900 32688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:52:27.854036 32688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:52:27.861761 32688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:52:27.861887 32688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:52:27.862051 32688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.129
I20260806 02:52:27.872638 32688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:40227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab
--principal=kudu/127.25.211.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.211.129
--webserver_port=0
--tserver_master_addrs=127.25.211.190:41447
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.129
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:52:27.874084 32688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:52:27.875516 32688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:52:27.888803 32696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:27.888837 32693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:27.888705 32694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:52:27.894418 32688 server_base.cc:1034] running on GCE node
I20260806 02:52:27.895093 32688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:52:27.896203 32688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:52:27.897431 32688 hybrid_clock.cc:648] HybridClock initialized: now 1785984747897383 us; error 38 us; skew 500 ppm
I20260806 02:52:27.897744 32688 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:52:27 dist-test-slave-zj8q krb5kdc[32523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984747, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:52:27.901170 32688 init.cc:377] Logged in from keytab as kudu/127.25.211.129@KRBTEST.COM (short username kudu)
I20260806 02:52:27.902514 32688 fs_manager.cc:359] Metadata directory not provided
I20260806 02:52:27.902629 32688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:52:27.902959 32688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:52:27.906016 32688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/ts-0/data/instance:
uuid: "fa3b27fb9fbc4f4eb8ac1c0dad5d6f0f"
format_stamp: "Formatted at 2026-08-06 02:52:27 on dist-test-slave-zj8q"
server_key: "e649135f3f77c1484cd727810f45da2c"
server_key_iv: "f9939a5fd1739c3c201bee3a03722618"
server_key_version: "encryptionkey@0"
I20260806 02:52:27.906905 32688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/ts-0/wal/instance:
uuid: "fa3b27fb9fbc4f4eb8ac1c0dad5d6f0f"
format_stamp: "Formatted at 2026-08-06 02:52:27 on dist-test-slave-zj8q"
server_key: "e649135f3f77c1484cd727810f45da2c"
server_key_iv: "f9939a5fd1739c3c201bee3a03722618"
server_key_version: "encryptionkey@0"
I20260806 02:52:27.912312 32688 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260806 02:52:27.916158 32705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:27.917577 32688 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:52:27.917776 32688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/ts-0/wal
uuid: "fa3b27fb9fbc4f4eb8ac1c0dad5d6f0f"
format_stamp: "Formatted at 2026-08-06 02:52:27 on dist-test-slave-zj8q"
server_key: "e649135f3f77c1484cd727810f45da2c"
server_key_iv: "f9939a5fd1739c3c201bee3a03722618"
server_key_version: "encryptionkey@0"
I20260806 02:52:27.918018 32688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:52:27.937465 32688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:52:27.942124 32688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:52:27.942463 32688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:52:27.943552 32688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:52:27.945124 32688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:52:27.945240 32688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:27.945356 32688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:52:27.945426 32688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:27.977860 32688 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.129:42035
I20260806 02:52:27.977912 405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.129:42035 every 8 connection(s)
I20260806 02:52:27.978789 32688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
I20260806 02:52:27.980628 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 32688
I20260806 02:52:27.980871 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/ts-0/wal/instance
I20260806 02:52:27.981611 26446 external_mini_cluster.cc:1428] Setting key cc633975155deb6266fd0dab256ff006
Aug 06 02:52:27 dist-test-slave-zj8q krb5kdc[32523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984747, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.129@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:52:28.003088 408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:27.981579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:46299 (local address 127.25.211.190:41447)
0806 02:52:27.982567 (+ 988us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:27.982575 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:27.983218 (+ 643us) server_negotiation.cc:407] Connection header received
0806 02:52:27.985189 (+ 1971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:27.985195 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:27.985285 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:27.985387 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:27.987178 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:27.987934 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:27.989704 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:27.990216 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:27.990338 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:27.994082 (+ 3744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:27.994090 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:27.994094 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:27.994130 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:27.999640 (+ 5510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:28.000520 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:28.000525 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:28.000527 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:28.000666 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:28.001137 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:28.001141 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:28.001143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:28.001436 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:28.001582 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:28.002436 (+ 854us) server_negotiation.cc:299] Negotiation successful
0806 02:52:28.002709 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":828,"thread_start_us":1225,"threads_started":1}
WARNING: no policy specified for kudu/127.25.211.130@KRBTEST.COM; defaulting to no policy
I20260806 02:52:28.004195 409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:27.982094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:41447 (local address 127.25.211.129:46299)
0806 02:52:27.983052 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:27.983100 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:27.984898 (+ 1798us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:27.985686 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:27.985696 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:27.986334 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:27.986978 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:27.986998 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:27.988110 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:27.988116 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:27.989534 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:27.989549 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:27.990460 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:27.990468 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:27.990706 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:27.991384 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:52:27.991408 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:27.993839 (+ 2431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:27.999834 (+ 5995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:27.999846 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:27.999864 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:28.000383 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:28.000786 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:28.000792 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:28.000796 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:28.001014 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:28.001595 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:28.001604 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:28.001993 (+ 389us) client_negotiation.cc:769] Sending connection context
0806 02:52:28.002314 (+ 321us) client_negotiation.cc:240] Negotiation successful
0806 02:52:28.002942 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"thread_start_us":382,"threads_started":1}
I20260806 02:52:28.005976 406 heartbeater.cc:347] Connected to a master server at 127.25.211.190:41447
I20260806 02:52:28.006393 406 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.130@KRBTEST.COM" created.
I20260806 02:52:28.007220 406 heartbeater.cc:511] Master 127.25.211.190:41447 requested a full tablet report, sending...
I20260806 02:52:28.009936 32587 ts_manager.cc:194] Registered new tserver with Master: fa3b27fb9fbc4f4eb8ac1c0dad5d6f0f (127.25.211.129:42035)
I20260806 02:52:28.011951 32587 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.211.129@KRBTEST.COM'} at 127.25.211.129:46299
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab.
Entry for principal kudu/127.25.211.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab.
I20260806 02:52:28.039152 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.130:0
--local_ip_for_outbound_sockets=127.25.211.130
--webserver_interface=127.25.211.130
--webserver_port=0
--tserver_master_addrs=127.25.211.190:41447
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab
--principal=kudu/127.25.211.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:40227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:52:28.383075 413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:52:28.383415 413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:52:28.383565 413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:52:28.391202 413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:52:28.391338 413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:52:28.391494 413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.130
I20260806 02:52:28.402105 413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:40227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab
--principal=kudu/127.25.211.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.211.130
--webserver_port=0
--tserver_master_addrs=127.25.211.190:41447
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.130
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:52:28.403566 413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:52:28.404920 413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:52:28.419212 422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:28.419212 420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:28.419373 419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:52:28.420528 413 server_base.cc:1034] running on GCE node
I20260806 02:52:28.421284 413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:52:28.422467 413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:52:28.423719 413 hybrid_clock.cc:648] HybridClock initialized: now 1785984748423640 us; error 68 us; skew 500 ppm
I20260806 02:52:28.424072 413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:52:28 dist-test-slave-zj8q krb5kdc[32523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984748, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:52:28.427549 413 init.cc:377] Logged in from keytab as kudu/127.25.211.130@KRBTEST.COM (short username kudu)
I20260806 02:52:28.428862 413 fs_manager.cc:359] Metadata directory not provided
I20260806 02:52:28.428963 413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:52:28.429292 413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:52:28.432878 413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/ts-1/data/instance:
uuid: "c33b1c72739e47f994b5a8f4597b84d1"
format_stamp: "Formatted at 2026-08-06 02:52:28 on dist-test-slave-zj8q"
server_key: "bca39139de2bc1440b7fb932ce29fef8"
server_key_iv: "ed9d33963848881b251ff88eeb25850e"
server_key_version: "encryptionkey@0"
I20260806 02:52:28.433799 413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/ts-1/wal/instance:
uuid: "c33b1c72739e47f994b5a8f4597b84d1"
format_stamp: "Formatted at 2026-08-06 02:52:28 on dist-test-slave-zj8q"
server_key: "bca39139de2bc1440b7fb932ce29fef8"
server_key_iv: "ed9d33963848881b251ff88eeb25850e"
server_key_version: "encryptionkey@0"
I20260806 02:52:28.439565 413 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260806 02:52:28.443470 430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:28.444903 413 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:52:28.445096 413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/ts-1/wal
uuid: "c33b1c72739e47f994b5a8f4597b84d1"
format_stamp: "Formatted at 2026-08-06 02:52:28 on dist-test-slave-zj8q"
server_key: "bca39139de2bc1440b7fb932ce29fef8"
server_key_iv: "ed9d33963848881b251ff88eeb25850e"
server_key_version: "encryptionkey@0"
I20260806 02:52:28.445348 413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:52:28.462971 413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:52:28.467041 413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:52:28.467298 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:52:28.468269 413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:52:28.469614 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:52:28.469703 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:28.469786 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:52:28.469854 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:28.500629 413 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.130:46221
I20260806 02:52:28.500716 591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.130:46221 every 8 connection(s)
I20260806 02:52:28.501493 413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
I20260806 02:52:28.504010 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 413
I20260806 02:52:28.504251 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/ts-1/wal/instance
I20260806 02:52:28.504935 26446 external_mini_cluster.cc:1428] Setting key 9689bb13f401eb6e21559318e403d4d2
Aug 06 02:52:28 dist-test-slave-zj8q krb5kdc[32523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984748, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.130@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:52:28.523456 596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:28.504012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:46595 (local address 127.25.211.190:41447)
0806 02:52:28.506522 (+ 2510us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:28.506526 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:28.506544 (+ 18us) server_negotiation.cc:407] Connection header received
0806 02:52:28.508085 (+ 1541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:28.508092 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:28.508180 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:28.508295 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:28.509939 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:28.510936 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:28.512562 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:28.513137 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:28.513258 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:28.517082 (+ 3824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:28.517088 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:28.517091 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:28.517128 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:28.519769 (+ 2641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:28.520596 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:28.520600 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:28.520603 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:28.520744 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:28.521191 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:28.521197 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:28.521200 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:28.521616 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:28.521761 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:28.522789 (+ 1028us) server_negotiation.cc:299] Negotiation successful
0806 02:52:28.523050 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2403,"thread_start_us":2304,"threads_started":1}
I20260806 02:52:28.524575 597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:28.504519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:41447 (local address 127.25.211.130:46595)
0806 02:52:28.505608 (+ 1089us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:28.505660 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:28.507796 (+ 2136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:28.508509 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:28.508518 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:28.509111 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:28.509722 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:28.509745 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:28.511081 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:28.511085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:28.512387 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:28.512402 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:28.513364 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:28.513368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:28.513575 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:28.514354 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:52:28.514379 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:28.516808 (+ 2429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:28.519939 (+ 3131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:28.519950 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:28.519968 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:28.520462 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:28.520860 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:28.520864 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:28.520868 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:28.521075 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:28.521757 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:28.521766 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:28.522268 (+ 502us) client_negotiation.cc:769] Sending connection context
0806 02:52:28.522682 (+ 414us) client_negotiation.cc:240] Negotiation successful
0806 02:52:28.523301 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":780,"thread_start_us":563,"threads_started":1}
I20260806 02:52:28.526434 592 heartbeater.cc:347] Connected to a master server at 127.25.211.190:41447
I20260806 02:52:28.526858 592 heartbeater.cc:464] Registering TS with master...
I20260806 02:52:28.527834 592 heartbeater.cc:511] Master 127.25.211.190:41447 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.25.211.131@KRBTEST.COM; defaulting to no policy
I20260806 02:52:28.529623 32587 ts_manager.cc:194] Registered new tserver with Master: c33b1c72739e47f994b5a8f4597b84d1 (127.25.211.130:46221)
I20260806 02:52:28.530516 32587 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.211.130@KRBTEST.COM'} at 127.25.211.130:46595
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab.
Entry for principal kudu/127.25.211.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab.
I20260806 02:52:28.561614 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.131:0
--local_ip_for_outbound_sockets=127.25.211.131
--webserver_interface=127.25.211.131
--webserver_port=0
--tserver_master_addrs=127.25.211.190:41447
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab
--principal=kudu/127.25.211.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:40227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:52:28.908275 601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:52:28.908589 601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:52:28.908717 601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:52:28.916278 601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:52:28.916410 601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:52:28.916564 601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.131
I20260806 02:52:28.927064 601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:40227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab
--principal=kudu/127.25.211.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.211.131
--webserver_port=0
--tserver_master_addrs=127.25.211.190:41447
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.131
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:52:28.928450 601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:52:28.929808 601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:52:28.943336 614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:28.943375 610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:28.943465 609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:52:28.944487 601 server_base.cc:1034] running on GCE node
I20260806 02:52:28.945235 601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:52:28.946377 601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:52:28.947602 601 hybrid_clock.cc:648] HybridClock initialized: now 1785984748947543 us; error 53 us; skew 500 ppm
I20260806 02:52:28.947952 601 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:52:28 dist-test-slave-zj8q krb5kdc[32523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984748, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:52:28.951436 601 init.cc:377] Logged in from keytab as kudu/127.25.211.131@KRBTEST.COM (short username kudu)
I20260806 02:52:28.952759 601 fs_manager.cc:359] Metadata directory not provided
I20260806 02:52:28.952875 601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:52:28.953207 601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:52:28.956501 601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/ts-2/data/instance:
uuid: "24836467eafc4f7a8e7a8ee64a1cb37e"
format_stamp: "Formatted at 2026-08-06 02:52:28 on dist-test-slave-zj8q"
server_key: "3da72f361897ba390deadcb25fbb298c"
server_key_iv: "f3b5aa6b678294d845859f420884f312"
server_key_version: "encryptionkey@0"
I20260806 02:52:28.957401 601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/ts-2/wal/instance:
uuid: "24836467eafc4f7a8e7a8ee64a1cb37e"
format_stamp: "Formatted at 2026-08-06 02:52:28 on dist-test-slave-zj8q"
server_key: "3da72f361897ba390deadcb25fbb298c"
server_key_iv: "f3b5aa6b678294d845859f420884f312"
server_key_version: "encryptionkey@0"
I20260806 02:52:28.963140 601 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260806 02:52:28.967207 623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:28.968788 601 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 02:52:28.968998 601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/ts-2/wal
uuid: "24836467eafc4f7a8e7a8ee64a1cb37e"
format_stamp: "Formatted at 2026-08-06 02:52:28 on dist-test-slave-zj8q"
server_key: "3da72f361897ba390deadcb25fbb298c"
server_key_iv: "f3b5aa6b678294d845859f420884f312"
server_key_version: "encryptionkey@0"
I20260806 02:52:28.969236 601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:52:28.981845 601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:52:28.985797 601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:52:28.986076 601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:52:28.987104 601 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:52:28.988485 601 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:52:28.988584 601 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:28.988699 601 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:52:28.988795 601 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:29.016270 406 heartbeater.cc:503] Master 127.25.211.190:41447 was elected leader, sending a full tablet report...
I20260806 02:52:29.021580 601 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.131:38739
I20260806 02:52:29.021663 783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.131:38739 every 8 connection(s)
I20260806 02:52:29.022557 601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
I20260806 02:52:29.026968 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 601
I20260806 02:52:29.027194 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/ts-2/wal/instance
I20260806 02:52:29.027733 26446 external_mini_cluster.cc:1428] Setting key 178d051c32bd901327c0f698759103a6
Aug 06 02:52:29 dist-test-slave-zj8q krb5kdc[32523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984748, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.131@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:52:29.045135 788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:29.025041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:39869 (local address 127.25.211.190:41447)
0806 02:52:29.025462 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:29.025467 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:29.026620 (+ 1153us) server_negotiation.cc:407] Connection header received
0806 02:52:29.029255 (+ 2635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:29.029260 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:29.029365 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:29.029490 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:29.031667 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:29.032451 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:29.034262 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:29.034758 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:29.034862 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:29.038793 (+ 3931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:29.038798 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:29.038802 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:29.038840 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:29.041509 (+ 2669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:29.042515 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:29.042520 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:29.042523 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:29.042716 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:29.043264 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:29.043268 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:29.043271 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:29.043644 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:29.043795 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:29.044654 (+ 859us) server_negotiation.cc:299] Negotiation successful
0806 02:52:29.044851 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":190,"threads_started":1}
I20260806 02:52:29.046232 789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:29.025652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:41447 (local address 127.25.211.131:39869)
0806 02:52:29.026428 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:29.026476 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:29.028917 (+ 2441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:29.029849 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:29.029862 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:29.030677 (+ 815us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:29.031413 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:29.031441 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:29.032665 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:29.032669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:29.034022 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:29.034044 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:29.034996 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:29.035001 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:29.035224 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:29.035942 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:52:29.035969 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:29.038459 (+ 2490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:29.041678 (+ 3219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:29.041691 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:29.041712 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:29.042370 (+ 658us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:29.042841 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:29.042846 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:29.042849 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:29.043132 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:29.043783 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:29.043789 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:29.044243 (+ 454us) client_negotiation.cc:769] Sending connection context
0806 02:52:29.044533 (+ 290us) client_negotiation.cc:240] Negotiation successful
0806 02:52:29.044963 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":273,"threads_started":1}
I20260806 02:52:29.047745 784 heartbeater.cc:347] Connected to a master server at 127.25.211.190:41447
I20260806 02:52:29.048069 784 heartbeater.cc:464] Registering TS with master...
I20260806 02:52:29.048812 784 heartbeater.cc:511] Master 127.25.211.190:41447 requested a full tablet report, sending...
I20260806 02:52:29.050287 32587 ts_manager.cc:194] Registered new tserver with Master: 24836467eafc4f7a8e7a8ee64a1cb37e (127.25.211.131:38739)
I20260806 02:52:29.051087 32587 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.211.131@KRBTEST.COM'} at 127.25.211.131:39869
I20260806 02:52:29.056915 26446 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:52:29.075371 788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:29.061884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43540 (local address 127.25.211.190:41447)
0806 02:52:29.062140 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:29.062148 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:29.062599 (+ 451us) server_negotiation.cc:407] Connection header received
0806 02:52:29.062779 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:29.062783 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:29.062889 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:29.062984 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:29.064717 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:29.065501 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:29.066908 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:29.067346 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:29.067448 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:29.069809 (+ 2361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:29.069814 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:29.069817 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:29.069854 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:29.072409 (+ 2555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:29.073144 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:29.073148 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:29.073151 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:29.073300 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:29.073769 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:29.073775 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:29.073778 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:29.074259 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:29.074474 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:29.074875 (+ 401us) server_negotiation.cc:299] Negotiation successful
0806 02:52:29.075078 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 02:52:29.079723 32587 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43540:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:52:29.087009 32587 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:52:29.113176 806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:29.100597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33410 (local address 127.25.211.130:46221)
0806 02:52:29.103189 (+ 2592us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:29.103200 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:29.103903 (+ 703us) server_negotiation.cc:407] Connection header received
0806 02:52:29.104376 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:29.104385 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:29.104658 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:29.104905 (+ 247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:52:29.105909 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:29.107755 (+ 1846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:29.110707 (+ 2952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:29.112022 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:29.112201 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:29.112394 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:29.112681 (+ 287us) server_negotiation.cc:299] Negotiation successful
0806 02:52:29.112871 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2392,"thread_start_us":2267,"threads_started":1}
I20260806 02:52:29.113301 803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:29.100430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.130:46221 (local address 127.0.0.1:33410)
0806 02:52:29.103737 (+ 3307us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:29.103777 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:29.104085 (+ 308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:29.105198 (+ 1113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:29.105205 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:29.105239 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:52:29.105685 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:29.105699 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:29.107950 (+ 2251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:29.107957 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:29.110506 (+ 2549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:29.110521 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:29.112225 (+ 1704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:29.112231 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:29.112431 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:29.112487 (+ 56us) client_negotiation.cc:769] Sending connection context
0806 02:52:29.112623 (+ 136us) client_negotiation.cc:240] Negotiation successful
0806 02:52:29.112838 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3032,"spinlock_wait_cycles":3328,"thread_start_us":2853,"threads_started":1}
I20260806 02:52:29.113744 808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:29.103209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.131:38739 (local address 127.0.0.1:53488)
0806 02:52:29.103866 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:29.103884 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:29.104085 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:29.105638 (+ 1553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:29.105644 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:29.105676 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:52:29.106177 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:29.106190 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:29.108337 (+ 2147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:29.108343 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:29.110761 (+ 2418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:29.110777 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:29.112758 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:29.112764 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:29.112928 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:29.112960 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 02:52:29.113060 (+ 100us) client_negotiation.cc:240] Negotiation successful
0806 02:52:29.113189 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"spinlock_wait_cycles":21248,"thread_start_us":1859,"threads_started":1}
I20260806 02:52:29.113911 810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:29.103864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53488 (local address 127.25.211.131:38739)
0806 02:52:29.104744 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:29.104754 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:29.104786 (+ 32us) server_negotiation.cc:407] Connection header received
0806 02:52:29.104887 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:29.104895 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:29.105123 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:29.105306 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:52:29.106435 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:29.108158 (+ 1723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:29.110952 (+ 2794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:29.112232 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:29.112429 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:29.112611 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:29.113406 (+ 795us) server_negotiation.cc:299] Negotiation successful
0806 02:52:29.113593 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":692,"thread_start_us":1049,"threads_started":1}
I20260806 02:52:29.118276 812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:29.104406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:42035 (local address 127.0.0.1:34074)
0806 02:52:29.110678 (+ 6272us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:29.110702 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:29.110844 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:29.111500 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:29.111505 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:29.111532 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:52:29.111896 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:29.111908 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:29.114281 (+ 2373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:29.114286 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:29.116083 (+ 1797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:29.116098 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:29.117436 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:29.117442 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:29.117621 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:29.117655 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 02:52:29.117740 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 02:52:29.117873 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6141,"thread_start_us":5992,"threads_started":1}
I20260806 02:52:29.118276 813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:29.104523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34074 (local address 127.25.211.129:42035)
0806 02:52:29.110215 (+ 5692us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:29.110226 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:29.110825 (+ 599us) server_negotiation.cc:407] Connection header received
0806 02:52:29.110985 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:29.110993 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:29.111274 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:29.111483 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:52:29.112086 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:29.113556 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:29.116251 (+ 2695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:29.117230 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:29.117435 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:29.117628 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:29.117796 (+ 168us) server_negotiation.cc:299] Negotiation successful
0806 02:52:29.117959 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5438,"thread_start_us":5464,"threads_started":1}
I20260806 02:52:29.118074 498 tablet_service.cc:1467] Processing CreateTablet for tablet cd329d0c9ba2456595015ec65312eb63 (DEFAULT_TABLE table=test-table [id=61368f4fb2a0419b8d67c614f15562d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:52:29.118075 692 tablet_service.cc:1467] Processing CreateTablet for tablet cd329d0c9ba2456595015ec65312eb63 (DEFAULT_TABLE table=test-table [id=61368f4fb2a0419b8d67c614f15562d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:52:29.119676 498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd329d0c9ba2456595015ec65312eb63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:29.119676 692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd329d0c9ba2456595015ec65312eb63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:29.121811 305 tablet_service.cc:1467] Processing CreateTablet for tablet cd329d0c9ba2456595015ec65312eb63 (DEFAULT_TABLE table=test-table [id=61368f4fb2a0419b8d67c614f15562d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:52:29.123126 305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd329d0c9ba2456595015ec65312eb63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:29.134967 814 tablet_bootstrap.cc:492] T cd329d0c9ba2456595015ec65312eb63 P fa3b27fb9fbc4f4eb8ac1c0dad5d6f0f: Bootstrap starting.
I20260806 02:52:29.137146 816 tablet_bootstrap.cc:492] T cd329d0c9ba2456595015ec65312eb63 P c33b1c72739e47f994b5a8f4597b84d1: Bootstrap starting.
I20260806 02:52:29.138841 814 tablet_bootstrap.cc:654] T cd329d0c9ba2456595015ec65312eb63 P fa3b27fb9fbc4f4eb8ac1c0dad5d6f0f: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:29.140146 814 log.cc:824] T cd329d0c9ba2456595015ec65312eb63 P fa3b27fb9fbc4f4eb8ac1c0dad5d6f0f: Log is configured to *not* fsync() on all Append() calls
I20260806 02:52:29.140846 816 tablet_bootstrap.cc:654] T cd329d0c9ba2456595015ec65312eb63 P c33b1c72739e47f994b5a8f4597b84d1: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:29.142215 816 log.cc:824] T cd329d0c9ba2456595015ec65312eb63 P c33b1c72739e47f994b5a8f4597b84d1: Log is configured to *not* fsync() on all Append() calls
I20260806 02:52:29.142272 818 tablet_bootstrap.cc:492] T cd329d0c9ba2456595015ec65312eb63 P 24836467eafc4f7a8e7a8ee64a1cb37e: Bootstrap starting.
I20260806 02:52:29.143590 814 tablet_bootstrap.cc:492] T cd329d0c9ba2456595015ec65312eb63 P fa3b27fb9fbc4f4eb8ac1c0dad5d6f0f: No bootstrap required, opened a new log
I20260806 02:52:29.143967 814 ts_tablet_manager.cc:1397] T cd329d0c9ba2456595015ec65312eb63 P fa3b27fb9fbc4f4eb8ac1c0dad5d6f0f: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.002s
I20260806 02:52:29.145367 818 tablet_bootstrap.cc:654] T cd329d0c9ba2456595015ec65312eb63 P 24836467eafc4f7a8e7a8ee64a1cb37e: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:29.146442 818 log.cc:824] T cd329d0c9ba2456595015ec65312eb63 P 24836467eafc4f7a8e7a8ee64a1cb37e: Log is configured to *not* fsync() on all Append() calls
I20260806 02:52:29.149127 818 tablet_bootstrap.cc:492] T cd329d0c9ba2456595015ec65312eb63 P 24836467eafc4f7a8e7a8ee64a1cb37e: No bootstrap required, opened a new log
I20260806 02:52:29.149431 818 ts_tablet_manager.cc:1397] T cd329d0c9ba2456595015ec65312eb63 P 24836467eafc4f7a8e7a8ee64a1cb37e: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260806 02:52:29.151567 816 tablet_bootstrap.cc:492] T cd329d0c9ba2456595015ec65312eb63 P c33b1c72739e47f994b5a8f4597b84d1: No bootstrap required, opened a new log
I20260806 02:52:29.151849 816 ts_tablet_manager.cc:1397] T cd329d0c9ba2456595015ec65312eb63 P c33b1c72739e47f994b5a8f4597b84d1: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20260806 02:52:29.154569 814 raft_consensus.cc:348] T cd329d0c9ba2456595015ec65312eb63 P fa3b27fb9fbc4f4eb8ac1c0dad5d6f0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c33b1c72739e47f994b5a8f4597b84d1" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46221 } } peers { permanent_uuid: "fa3b27fb9fbc4f4eb8ac1c0dad5d6f0f" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42035 } } peers { permanent_uuid: "24836467eafc4f7a8e7a8ee64a1cb37e" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38739 } }
I20260806 02:52:29.155059 814 raft_consensus.cc:374] T cd329d0c9ba2456595015ec65312eb63 P fa3b27fb9fbc4f4eb8ac1c0dad5d6f0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:29.155205 814 raft_consensus.cc:729] T cd329d0c9ba2456595015ec65312eb63 P fa3b27fb9fbc4f4eb8ac1c0dad5d6f0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa3b27fb9fbc4f4eb8ac1c0dad5d6f0f, State: Initialized, Role: FOLLOWER
I20260806 02:52:29.155891 814 consensus_queue.cc:260] T cd329d0c9ba2456595015ec65312eb63 P fa3b27fb9fbc4f4eb8ac1c0dad5d6f0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c33b1c72739e47f994b5a8f4597b84d1" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46221 } } peers { permanent_uuid: "fa3b27fb9fbc4f4eb8ac1c0dad5d6f0f" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42035 } } peers { permanent_uuid: "24836467eafc4f7a8e7a8ee64a1cb37e" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38739 } }
I20260806 02:52:29.157814 814 ts_tablet_manager.cc:1428] T cd329d0c9ba2456595015ec65312eb63 P fa3b27fb9fbc4f4eb8ac1c0dad5d6f0f: Time spent starting tablet: real 0.014s user 0.006s sys 0.006s
I20260806 02:52:29.158095 818 raft_consensus.cc:348] T cd329d0c9ba2456595015ec65312eb63 P 24836467eafc4f7a8e7a8ee64a1cb37e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c33b1c72739e47f994b5a8f4597b84d1" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46221 } } peers { permanent_uuid: "fa3b27fb9fbc4f4eb8ac1c0dad5d6f0f" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42035 } } peers { permanent_uuid: "24836467eafc4f7a8e7a8ee64a1cb37e" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38739 } }
I20260806 02:52:29.158586 818 raft_consensus.cc:374] T cd329d0c9ba2456595015ec65312eb63 P 24836467eafc4f7a8e7a8ee64a1cb37e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:29.158689 818 raft_consensus.cc:729] T cd329d0c9ba2456595015ec65312eb63 P 24836467eafc4f7a8e7a8ee64a1cb37e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24836467eafc4f7a8e7a8ee64a1cb37e, State: Initialized, Role: FOLLOWER
I20260806 02:52:29.159426 818 consensus_queue.cc:260] T cd329d0c9ba2456595015ec65312eb63 P 24836467eafc4f7a8e7a8ee64a1cb37e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c33b1c72739e47f994b5a8f4597b84d1" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46221 } } peers { permanent_uuid: "fa3b27fb9fbc4f4eb8ac1c0dad5d6f0f" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42035 } } peers { permanent_uuid: "24836467eafc4f7a8e7a8ee64a1cb37e" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38739 } }
I20260806 02:52:29.160648 784 heartbeater.cc:503] Master 127.25.211.190:41447 was elected leader, sending a full tablet report...
I20260806 02:52:29.161350 818 ts_tablet_manager.cc:1428] T cd329d0c9ba2456595015ec65312eb63 P 24836467eafc4f7a8e7a8ee64a1cb37e: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260806 02:52:29.161554 816 raft_consensus.cc:348] T cd329d0c9ba2456595015ec65312eb63 P c33b1c72739e47f994b5a8f4597b84d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c33b1c72739e47f994b5a8f4597b84d1" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46221 } } peers { permanent_uuid: "fa3b27fb9fbc4f4eb8ac1c0dad5d6f0f" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42035 } } peers { permanent_uuid: "24836467eafc4f7a8e7a8ee64a1cb37e" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38739 } }
I20260806 02:52:29.161996 816 raft_consensus.cc:374] T cd329d0c9ba2456595015ec65312eb63 P c33b1c72739e47f994b5a8f4597b84d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:29.162101 816 raft_consensus.cc:729] T cd329d0c9ba2456595015ec65312eb63 P c33b1c72739e47f994b5a8f4597b84d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c33b1c72739e47f994b5a8f4597b84d1, State: Initialized, Role: FOLLOWER
I20260806 02:52:29.162681 816 consensus_queue.cc:260] T cd329d0c9ba2456595015ec65312eb63 P c33b1c72739e47f994b5a8f4597b84d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c33b1c72739e47f994b5a8f4597b84d1" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46221 } } peers { permanent_uuid: "fa3b27fb9fbc4f4eb8ac1c0dad5d6f0f" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42035 } } peers { permanent_uuid: "24836467eafc4f7a8e7a8ee64a1cb37e" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38739 } }
I20260806 02:52:29.167006 592 heartbeater.cc:503] Master 127.25.211.190:41447 was elected leader, sending a full tablet report...
I20260806 02:52:29.167658 816 ts_tablet_manager.cc:1428] T cd329d0c9ba2456595015ec65312eb63 P c33b1c72739e47f994b5a8f4597b84d1: Time spent starting tablet: real 0.016s user 0.009s sys 0.007s
I20260806 02:52:29.169728 827 raft_consensus.cc:482] T cd329d0c9ba2456595015ec65312eb63 P 24836467eafc4f7a8e7a8ee64a1cb37e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:52:29.170002 827 raft_consensus.cc:504] T cd329d0c9ba2456595015ec65312eb63 P 24836467eafc4f7a8e7a8ee64a1cb37e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c33b1c72739e47f994b5a8f4597b84d1" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46221 } } peers { permanent_uuid: "fa3b27fb9fbc4f4eb8ac1c0dad5d6f0f" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42035 } } peers { permanent_uuid: "24836467eafc4f7a8e7a8ee64a1cb37e" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38739 } }
I20260806 02:52:29.171730 827 leader_election.cc:288] T cd329d0c9ba2456595015ec65312eb63 P 24836467eafc4f7a8e7a8ee64a1cb37e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c33b1c72739e47f994b5a8f4597b84d1 (127.25.211.130:46221), fa3b27fb9fbc4f4eb8ac1c0dad5d6f0f (127.25.211.129:42035)
I20260806 02:52:29.179934 806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:29.173114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:55887 (local address 127.25.211.130:46221)
0806 02:52:29.173303 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:29.173308 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:29.173328 (+ 20us) server_negotiation.cc:407] Connection header received
0806 02:52:29.173395 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:29.173399 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:29.173458 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:29.173532 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:52:29.173963 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:29.175089 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:29.177605 (+ 2516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:29.178872 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:29.179045 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:29.179137 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:29.179526 (+ 389us) server_negotiation.cc:299] Negotiation successful
0806 02:52:29.179664 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 02:52:29.179935 813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:29.172784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:33555 (local address 127.25.211.129:42035)
0806 02:52:29.172991 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:29.172997 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:29.173208 (+ 211us) server_negotiation.cc:407] Connection header received
0806 02:52:29.173424 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:29.173428 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:29.173484 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:29.173564 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:52:29.174273 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:29.175329 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:29.177737 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:29.178930 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:29.179065 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:29.179134 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:29.179526 (+ 392us) server_negotiation.cc:299] Negotiation successful
0806 02:52:29.179697 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 02:52:29.180111 829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:29.172520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:42035 (local address 127.25.211.131:33555)
0806 02:52:29.173117 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:29.173136 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:29.173296 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:29.173609 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:29.173615 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:29.173672 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:52:29.174064 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:29.174076 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:29.175470 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:29.175474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:29.177581 (+ 2107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:29.177593 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:29.179120 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:29.179126 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:29.179330 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:29.179387 (+ 57us) client_negotiation.cc:769] Sending connection context
0806 02:52:29.179479 (+ 92us) client_negotiation.cc:240] Negotiation successful
0806 02:52:29.179667 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"mutex_wait_us":63,"spinlock_wait_cycles":2176,"thread_start_us":651,"threads_started":1}
I20260806 02:52:29.180073 789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:29.172133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.130:46221 (local address 127.25.211.131:55887)
0806 02:52:29.172330 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:29.172347 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:29.172441 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:29.173572 (+ 1131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:29.173576 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:29.173602 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:52:29.173819 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:29.173832 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:29.175239 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:29.175243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:29.177399 (+ 2156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:29.177418 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:29.179043 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:29.179050 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:29.179289 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:29.179383 (+ 94us) client_negotiation.cc:769] Sending connection context
0806 02:52:29.179479 (+ 96us) client_negotiation.cc:240] Negotiation successful
0806 02:52:29.179667 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":64,"spinlock_wait_cycles":4352}
I20260806 02:52:29.181282 338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd329d0c9ba2456595015ec65312eb63" candidate_uuid: "24836467eafc4f7a8e7a8ee64a1cb37e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa3b27fb9fbc4f4eb8ac1c0dad5d6f0f" is_pre_election: true
I20260806 02:52:29.181484 527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd329d0c9ba2456595015ec65312eb63" candidate_uuid: "24836467eafc4f7a8e7a8ee64a1cb37e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c33b1c72739e47f994b5a8f4597b84d1" is_pre_election: true
I20260806 02:52:29.181826 338 raft_consensus.cc:2454] T cd329d0c9ba2456595015ec65312eb63 P fa3b27fb9fbc4f4eb8ac1c0dad5d6f0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24836467eafc4f7a8e7a8ee64a1cb37e in term 0.
I20260806 02:52:29.181999 527 raft_consensus.cc:2454] T cd329d0c9ba2456595015ec65312eb63 P c33b1c72739e47f994b5a8f4597b84d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24836467eafc4f7a8e7a8ee64a1cb37e in term 0.
I20260806 02:52:29.182536 629 leader_election.cc:302] T cd329d0c9ba2456595015ec65312eb63 P 24836467eafc4f7a8e7a8ee64a1cb37e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24836467eafc4f7a8e7a8ee64a1cb37e, fa3b27fb9fbc4f4eb8ac1c0dad5d6f0f; no voters:
I20260806 02:52:29.182986 827 raft_consensus.cc:2781] T cd329d0c9ba2456595015ec65312eb63 P 24836467eafc4f7a8e7a8ee64a1cb37e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:52:29.183171 827 raft_consensus.cc:482] T cd329d0c9ba2456595015ec65312eb63 P 24836467eafc4f7a8e7a8ee64a1cb37e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:52:29.183298 827 raft_consensus.cc:3037] T cd329d0c9ba2456595015ec65312eb63 P 24836467eafc4f7a8e7a8ee64a1cb37e [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:29.185822 827 raft_consensus.cc:504] T cd329d0c9ba2456595015ec65312eb63 P 24836467eafc4f7a8e7a8ee64a1cb37e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c33b1c72739e47f994b5a8f4597b84d1" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46221 } } peers { permanent_uuid: "fa3b27fb9fbc4f4eb8ac1c0dad5d6f0f" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42035 } } peers { permanent_uuid: "24836467eafc4f7a8e7a8ee64a1cb37e" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38739 } }
I20260806 02:52:29.186506 827 leader_election.cc:288] T cd329d0c9ba2456595015ec65312eb63 P 24836467eafc4f7a8e7a8ee64a1cb37e [CANDIDATE]: Term 1 election: Requested vote from peers c33b1c72739e47f994b5a8f4597b84d1 (127.25.211.130:46221), fa3b27fb9fbc4f4eb8ac1c0dad5d6f0f (127.25.211.129:42035)
I20260806 02:52:29.187115 527 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd329d0c9ba2456595015ec65312eb63" candidate_uuid: "24836467eafc4f7a8e7a8ee64a1cb37e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c33b1c72739e47f994b5a8f4597b84d1"
I20260806 02:52:29.187160 338 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd329d0c9ba2456595015ec65312eb63" candidate_uuid: "24836467eafc4f7a8e7a8ee64a1cb37e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa3b27fb9fbc4f4eb8ac1c0dad5d6f0f"
I20260806 02:52:29.187350 527 raft_consensus.cc:3037] T cd329d0c9ba2456595015ec65312eb63 P c33b1c72739e47f994b5a8f4597b84d1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:29.187397 338 raft_consensus.cc:3037] T cd329d0c9ba2456595015ec65312eb63 P fa3b27fb9fbc4f4eb8ac1c0dad5d6f0f [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:29.189910 338 raft_consensus.cc:2454] T cd329d0c9ba2456595015ec65312eb63 P fa3b27fb9fbc4f4eb8ac1c0dad5d6f0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24836467eafc4f7a8e7a8ee64a1cb37e in term 1.
I20260806 02:52:29.189913 527 raft_consensus.cc:2454] T cd329d0c9ba2456595015ec65312eb63 P c33b1c72739e47f994b5a8f4597b84d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24836467eafc4f7a8e7a8ee64a1cb37e in term 1.
I20260806 02:52:29.190501 629 leader_election.cc:302] T cd329d0c9ba2456595015ec65312eb63 P 24836467eafc4f7a8e7a8ee64a1cb37e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24836467eafc4f7a8e7a8ee64a1cb37e, fa3b27fb9fbc4f4eb8ac1c0dad5d6f0f; no voters:
I20260806 02:52:29.190847 827 raft_consensus.cc:2781] T cd329d0c9ba2456595015ec65312eb63 P 24836467eafc4f7a8e7a8ee64a1cb37e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:52:29.191342 827 raft_consensus.cc:686] T cd329d0c9ba2456595015ec65312eb63 P 24836467eafc4f7a8e7a8ee64a1cb37e [term 1 LEADER]: Becoming Leader. State: Replica: 24836467eafc4f7a8e7a8ee64a1cb37e, State: Running, Role: LEADER
I20260806 02:52:29.191874 827 consensus_queue.cc:237] T cd329d0c9ba2456595015ec65312eb63 P 24836467eafc4f7a8e7a8ee64a1cb37e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c33b1c72739e47f994b5a8f4597b84d1" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46221 } } peers { permanent_uuid: "fa3b27fb9fbc4f4eb8ac1c0dad5d6f0f" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42035 } } peers { permanent_uuid: "24836467eafc4f7a8e7a8ee64a1cb37e" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38739 } }
I20260806 02:52:29.197124 32585 catalog_manager.cc:5495] T cd329d0c9ba2456595015ec65312eb63 P 24836467eafc4f7a8e7a8ee64a1cb37e reported cstate change: term changed from 0 to 1, leader changed from <none> to 24836467eafc4f7a8e7a8ee64a1cb37e (127.25.211.131). New cstate: current_term: 1 leader_uuid: "24836467eafc4f7a8e7a8ee64a1cb37e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c33b1c72739e47f994b5a8f4597b84d1" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa3b27fb9fbc4f4eb8ac1c0dad5d6f0f" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24836467eafc4f7a8e7a8ee64a1cb37e" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38739 } health_report { overall_health: HEALTHY } } }
I20260806 02:52:29.216571 810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:29.210971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53500 (local address 127.25.211.131:38739)
0806 02:52:29.211219 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:29.211225 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:29.211247 (+ 22us) server_negotiation.cc:407] Connection header received
0806 02:52:29.211423 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:29.211428 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:29.211511 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:29.211664 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:52:29.212069 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:29.212815 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:29.214294 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:29.214736 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:29.214858 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:29.215387 (+ 529us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:52:29.215959 (+ 572us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:52:29.216097 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:29.216211 (+ 114us) server_negotiation.cc:299] Negotiation successful
0806 02:52:29.216302 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260806 02:52:29.224867 338 raft_consensus.cc:1261] T cd329d0c9ba2456595015ec65312eb63 P fa3b27fb9fbc4f4eb8ac1c0dad5d6f0f [term 1 FOLLOWER]: Refusing update from remote peer 24836467eafc4f7a8e7a8ee64a1cb37e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:52:29.224932 527 raft_consensus.cc:1261] T cd329d0c9ba2456595015ec65312eb63 P c33b1c72739e47f994b5a8f4597b84d1 [term 1 FOLLOWER]: Refusing update from remote peer 24836467eafc4f7a8e7a8ee64a1cb37e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:52:29.225863 830 consensus_queue.cc:1035] T cd329d0c9ba2456595015ec65312eb63 P 24836467eafc4f7a8e7a8ee64a1cb37e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c33b1c72739e47f994b5a8f4597b84d1" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 46221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:52:29.226292 827 consensus_queue.cc:1035] T cd329d0c9ba2456595015ec65312eb63 P 24836467eafc4f7a8e7a8ee64a1cb37e [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa3b27fb9fbc4f4eb8ac1c0dad5d6f0f" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 02:52:29.231120 407 tablet.cc:2369] T cd329d0c9ba2456595015ec65312eb63 P fa3b27fb9fbc4f4eb8ac1c0dad5d6f0f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:52:29.269065 32587 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:52:29.269452 32587 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:52:29.273908 32587 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c34a097779344299be8a6b87d6a1c232: Sending DeleteTablet for 3 replicas of tablet cd329d0c9ba2456595015ec65312eb63
I20260806 02:52:29.275432 498 tablet_service.cc:1514] Processing DeleteTablet for tablet cd329d0c9ba2456595015ec65312eb63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:52:29 UTC) from {username='kudu', principal='kudu/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:33410
I20260806 02:52:29.276008 692 tablet_service.cc:1514] Processing DeleteTablet for tablet cd329d0c9ba2456595015ec65312eb63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:52:29 UTC) from {username='kudu', principal='kudu/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:53488
I20260806 02:52:29.276001 305 tablet_service.cc:1514] Processing DeleteTablet for tablet cd329d0c9ba2456595015ec65312eb63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:52:29 UTC) from {username='kudu', principal='kudu/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:34074
I20260806 02:52:29.279959 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 32688
I20260806 02:52:29.296042 850 tablet_replica.cc:330] T cd329d0c9ba2456595015ec65312eb63 P fa3b27fb9fbc4f4eb8ac1c0dad5d6f0f: stopping tablet replica
I20260806 02:52:29.295507 849 tablet_replica.cc:330] T cd329d0c9ba2456595015ec65312eb63 P 24836467eafc4f7a8e7a8ee64a1cb37e: stopping tablet replica
I20260806 02:52:29.297169 849 raft_consensus.cc:2229] T cd329d0c9ba2456595015ec65312eb63 P 24836467eafc4f7a8e7a8ee64a1cb37e [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:52:29.297863 849 raft_consensus.cc:2258] T cd329d0c9ba2456595015ec65312eb63 P 24836467eafc4f7a8e7a8ee64a1cb37e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:52:29.296571 850 raft_consensus.cc:2229] T cd329d0c9ba2456595015ec65312eb63 P fa3b27fb9fbc4f4eb8ac1c0dad5d6f0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:52:29.298854 850 raft_consensus.cc:2258] T cd329d0c9ba2456595015ec65312eb63 P fa3b27fb9fbc4f4eb8ac1c0dad5d6f0f [term 1 FOLLOWER]: Raft consensus is shut down!
Aug 06 02:52:29 dist-test-slave-zj8q krb5kdc[32523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.129@KRBTEST.COM
I20260806 02:52:29.301153 850 ts_tablet_manager.cc:1905] T cd329d0c9ba2456595015ec65312eb63 P fa3b27fb9fbc4f4eb8ac1c0dad5d6f0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:52:29.302069 848 tablet_replica.cc:330] T cd329d0c9ba2456595015ec65312eb63 P c33b1c72739e47f994b5a8f4597b84d1: stopping tablet replica
I20260806 02:52:29.302567 848 raft_consensus.cc:2229] T cd329d0c9ba2456595015ec65312eb63 P c33b1c72739e47f994b5a8f4597b84d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:52:29.303154 848 raft_consensus.cc:2258] T cd329d0c9ba2456595015ec65312eb63 P c33b1c72739e47f994b5a8f4597b84d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:52:29.306739 848 ts_tablet_manager.cc:1905] T cd329d0c9ba2456595015ec65312eb63 P c33b1c72739e47f994b5a8f4597b84d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:52:29.307724 850 ts_tablet_manager.cc:1918] T cd329d0c9ba2456595015ec65312eb63 P fa3b27fb9fbc4f4eb8ac1c0dad5d6f0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:52:29.307920 850 log.cc:1196] T cd329d0c9ba2456595015ec65312eb63 P fa3b27fb9fbc4f4eb8ac1c0dad5d6f0f: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/ts-0/wal/wals/cd329d0c9ba2456595015ec65312eb63
I20260806 02:52:29.308460 850 ts_tablet_manager.cc:1939] T cd329d0c9ba2456595015ec65312eb63 P fa3b27fb9fbc4f4eb8ac1c0dad5d6f0f: Deleting consensus metadata
I20260806 02:52:29.310431 32567 catalog_manager.cc:4841] TS fa3b27fb9fbc4f4eb8ac1c0dad5d6f0f (127.25.211.129:42035): tablet cd329d0c9ba2456595015ec65312eb63 (table test-table [id=61368f4fb2a0419b8d67c614f15562d6]) successfully deleted
I20260806 02:52:29.311815 849 ts_tablet_manager.cc:1905] T cd329d0c9ba2456595015ec65312eb63 P 24836467eafc4f7a8e7a8ee64a1cb37e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:52:29.313995 813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:29.282529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34084 (local address 127.25.211.129:42035)
0806 02:52:29.282765 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:29.282773 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:29.284016 (+ 1243us) server_negotiation.cc:407] Connection header received
0806 02:52:29.284960 (+ 944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:29.284967 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:29.285073 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:29.285152 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:29.288878 (+ 3726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:29.290079 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:29.293852 (+ 3773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:29.294655 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:29.294764 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:29.304929 (+ 10165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:29.304947 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:29.304966 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:29.305021 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:29.309192 (+ 4171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:29.310253 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:29.310264 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:29.310268 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:29.310461 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:29.310978 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:29.310984 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:29.310986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:29.311505 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:29.311818 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:29.313379 (+ 1561us) server_negotiation.cc:299] Negotiation successful
0806 02:52:29.313625 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 02:52:29.314929 400 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:52:29.316164 848 ts_tablet_manager.cc:1918] T cd329d0c9ba2456595015ec65312eb63 P c33b1c72739e47f994b5a8f4597b84d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:52:29.316354 848 log.cc:1196] T cd329d0c9ba2456595015ec65312eb63 P c33b1c72739e47f994b5a8f4597b84d1: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/ts-1/wal/wals/cd329d0c9ba2456595015ec65312eb63
I20260806 02:52:29.316915 848 ts_tablet_manager.cc:1939] T cd329d0c9ba2456595015ec65312eb63 P c33b1c72739e47f994b5a8f4597b84d1: Deleting consensus metadata
I20260806 02:52:29.318956 32567 catalog_manager.cc:4841] TS c33b1c72739e47f994b5a8f4597b84d1 (127.25.211.130:46221): tablet cd329d0c9ba2456595015ec65312eb63 (table test-table [id=61368f4fb2a0419b8d67c614f15562d6]) successfully deleted
I20260806 02:52:29.320233 849 ts_tablet_manager.cc:1918] T cd329d0c9ba2456595015ec65312eb63 P 24836467eafc4f7a8e7a8ee64a1cb37e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:52:29.320436 849 log.cc:1196] T cd329d0c9ba2456595015ec65312eb63 P 24836467eafc4f7a8e7a8ee64a1cb37e: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1785984707115702-26446-0/minicluster-data/ts-2/wal/wals/cd329d0c9ba2456595015ec65312eb63
I20260806 02:52:29.321000 849 ts_tablet_manager.cc:1939] T cd329d0c9ba2456595015ec65312eb63 P 24836467eafc4f7a8e7a8ee64a1cb37e: Deleting consensus metadata
I20260806 02:52:29.322896 32562 catalog_manager.cc:4841] TS 24836467eafc4f7a8e7a8ee64a1cb37e (127.25.211.131:38739): tablet cd329d0c9ba2456595015ec65312eb63 (table test-table [id=61368f4fb2a0419b8d67c614f15562d6]) successfully deleted
I20260806 02:52:29.436414 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 32688
I20260806 02:52:29.461649 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 413
Aug 06 02:52:29 dist-test-slave-zj8q krb5kdc[32523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.130@KRBTEST.COM
I20260806 02:52:29.477571 806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:29.462762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33424 (local address 127.25.211.130:46221)
0806 02:52:29.462944 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:29.462948 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:29.462963 (+ 15us) server_negotiation.cc:407] Connection header received
0806 02:52:29.463018 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:29.463022 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:29.463083 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:29.463174 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:29.464457 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:29.465226 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:29.466721 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:29.467123 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:29.467237 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:29.471517 (+ 4280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:29.471534 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:29.471551 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:29.471594 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:29.474724 (+ 3130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:29.475471 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:29.475479 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:29.475481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:29.475626 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:29.475990 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:29.475994 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:29.475996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:29.476401 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:29.476695 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:29.477001 (+ 306us) server_negotiation.cc:299] Negotiation successful
0806 02:52:29.477239 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 02:52:29.478307 582 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:52:29.569428 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 413
I20260806 02:52:29.593398 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 601
Aug 06 02:52:29 dist-test-slave-zj8q krb5kdc[32523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.131@KRBTEST.COM
I20260806 02:52:29.610886 810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:29.594242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53506 (local address 127.25.211.131:38739)
0806 02:52:29.594439 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:29.594445 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:29.594527 (+ 82us) server_negotiation.cc:407] Connection header received
0806 02:52:29.594723 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:29.594729 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:29.594808 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:29.594903 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:29.596240 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:29.597124 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:29.598624 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:29.599085 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:29.599200 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:29.604156 (+ 4956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:29.604175 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:29.604196 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:29.604266 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:29.607688 (+ 3422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:29.608544 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:29.608553 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:29.608555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:29.608744 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:29.609153 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:29.609157 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:29.609159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:29.609543 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:29.609871 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:29.610228 (+ 357us) server_negotiation.cc:299] Negotiation successful
0806 02:52:29.610460 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260806 02:52:29.611768 778 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:52:29.737831 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 601
I20260806 02:52:29.762319 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 32541
I20260806 02:52:29.763563 32641 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:52:29.872905 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 32541
2026-08-06T02:52:29Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4457 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-06T02:52:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:52:29Z Disabled control of system clock
I20260806 02:52:29.952960 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:43587
--webserver_interface=127.25.211.190
--webserver_port=0
--builtin_ntp_servers=127.25.211.148:33715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.211.190:43587
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260806 02:52:30.297969 865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:52:30.298372 865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:52:30.298477 865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:52:30.307003 865 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:52:30.307152 865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:52:30.307216 865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:52:30.307301 865 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:52:30.307359 865 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:52:30.318142 865 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:33715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.211.190:43587
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:43587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.211.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:52:30.319595 865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:52:30.321623 865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:52:30.334506 873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:30.334475 877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:30.334424 879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:52:30.335512 865 server_base.cc:1034] running on GCE node
I20260806 02:52:30.336436 865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:52:30.338063 865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:52:30.339280 865 hybrid_clock.cc:648] HybridClock initialized: now 1785984750339232 us; error 42 us; skew 500 ppm
I20260806 02:52:30.339666 865 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:52:30.342334 865 webserver.cc:460] Webserver started at http://127.25.211.190:40203/ using document root <none> and password file <none>
I20260806 02:52:30.343128 865 fs_manager.cc:359] Metadata directory not provided
I20260806 02:52:30.343235 865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:52:30.343585 865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:52:30.346861 865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/master-0/data/instance:
uuid: "dae07894198141bfb955302402cfebec"
format_stamp: "Formatted at 2026-08-06 02:52:30 on dist-test-slave-zj8q"
server_key: "85a401f9def4c723c9b7e4e23656e48a"
server_key_iv: "d99dd414739fb772eb184a5feb6de953"
server_key_version: "encryptionkey@0"
I20260806 02:52:30.347728 865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/master-0/wal/instance:
uuid: "dae07894198141bfb955302402cfebec"
format_stamp: "Formatted at 2026-08-06 02:52:30 on dist-test-slave-zj8q"
server_key: "85a401f9def4c723c9b7e4e23656e48a"
server_key_iv: "d99dd414739fb772eb184a5feb6de953"
server_key_version: "encryptionkey@0"
I20260806 02:52:30.353343 865 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260806 02:52:30.357308 886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:30.358901 865 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:52:30.359091 865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/master-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/master-0/wal
uuid: "dae07894198141bfb955302402cfebec"
format_stamp: "Formatted at 2026-08-06 02:52:30 on dist-test-slave-zj8q"
server_key: "85a401f9def4c723c9b7e4e23656e48a"
server_key_iv: "d99dd414739fb772eb184a5feb6de953"
server_key_version: "encryptionkey@0"
I20260806 02:52:30.359328 865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:52:30.365571 865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:52:30.366526 865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:52:30.366835 865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:52:30.385268 865 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.190:43587
I20260806 02:52:30.385304 959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.190:43587 every 8 connection(s)
I20260806 02:52:30.386801 865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
I20260806 02:52:30.390863 960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:30.395143 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 865
I20260806 02:52:30.395330 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/master-0/wal/instance
I20260806 02:52:30.395980 26446 external_mini_cluster.cc:1428] Setting key af8e2bd3f4deed09e39dcec81c7ccea0
I20260806 02:52:30.404131 960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dae07894198141bfb955302402cfebec: Bootstrap starting.
I20260806 02:52:30.406560 966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:30.398244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57358 (local address 127.25.211.190:43587)
0806 02:52:30.399449 (+ 1205us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:30.399468 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:30.399526 (+ 58us) server_negotiation.cc:407] Connection header received
0806 02:52:30.402006 (+ 2480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:30.402051 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:30.402749 (+ 698us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:30.403252 (+ 503us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:30.404366 (+ 1114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:30.404375 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:30.404392 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:52:30.404411 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:30.404492 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:52:30.404516 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:30.404744 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:30.404879 (+ 135us) server_negotiation.cc:299] Negotiation successful
0806 02:52:30.405182 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":389,"threads_started":1}
I20260806 02:52:30.408142 960 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dae07894198141bfb955302402cfebec: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:30.409241 960 log.cc:824] T 00000000000000000000000000000000 P dae07894198141bfb955302402cfebec: Log is configured to *not* fsync() on all Append() calls
I20260806 02:52:30.412103 960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dae07894198141bfb955302402cfebec: No bootstrap required, opened a new log
I20260806 02:52:30.419350 960 raft_consensus.cc:348] T 00000000000000000000000000000000 P dae07894198141bfb955302402cfebec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dae07894198141bfb955302402cfebec" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 43587 } }
I20260806 02:52:30.419742 960 raft_consensus.cc:374] T 00000000000000000000000000000000 P dae07894198141bfb955302402cfebec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:30.419835 960 raft_consensus.cc:729] T 00000000000000000000000000000000 P dae07894198141bfb955302402cfebec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dae07894198141bfb955302402cfebec, State: Initialized, Role: FOLLOWER
I20260806 02:52:30.420390 960 consensus_queue.cc:260] T 00000000000000000000000000000000 P dae07894198141bfb955302402cfebec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dae07894198141bfb955302402cfebec" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 43587 } }
I20260806 02:52:30.420622 960 raft_consensus.cc:388] T 00000000000000000000000000000000 P dae07894198141bfb955302402cfebec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:52:30.420751 960 raft_consensus.cc:482] T 00000000000000000000000000000000 P dae07894198141bfb955302402cfebec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:52:30.420912 960 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dae07894198141bfb955302402cfebec [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:30.423316 960 raft_consensus.cc:504] T 00000000000000000000000000000000 P dae07894198141bfb955302402cfebec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dae07894198141bfb955302402cfebec" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 43587 } }
I20260806 02:52:30.423877 960 leader_election.cc:302] T 00000000000000000000000000000000 P dae07894198141bfb955302402cfebec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dae07894198141bfb955302402cfebec; no voters:
I20260806 02:52:30.424465 960 leader_election.cc:288] T 00000000000000000000000000000000 P dae07894198141bfb955302402cfebec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:52:30.424672 968 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dae07894198141bfb955302402cfebec [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:52:30.425370 968 raft_consensus.cc:686] T 00000000000000000000000000000000 P dae07894198141bfb955302402cfebec [term 1 LEADER]: Becoming Leader. State: Replica: dae07894198141bfb955302402cfebec, State: Running, Role: LEADER
I20260806 02:52:30.426003 968 consensus_queue.cc:237] T 00000000000000000000000000000000 P dae07894198141bfb955302402cfebec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dae07894198141bfb955302402cfebec" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 43587 } }
I20260806 02:52:30.426852 960 sys_catalog.cc:564] T 00000000000000000000000000000000 P dae07894198141bfb955302402cfebec [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:52:30.429360 971 sys_catalog.cc:455] T 00000000000000000000000000000000 P dae07894198141bfb955302402cfebec [sys.catalog]: SysCatalogTable state changed. Reason: New leader dae07894198141bfb955302402cfebec. Latest consensus state: current_term: 1 leader_uuid: "dae07894198141bfb955302402cfebec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dae07894198141bfb955302402cfebec" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 43587 } } }
I20260806 02:52:30.429790 971 sys_catalog.cc:458] T 00000000000000000000000000000000 P dae07894198141bfb955302402cfebec [sys.catalog]: This master's current role is: LEADER
I20260806 02:52:30.429616 970 sys_catalog.cc:455] T 00000000000000000000000000000000 P dae07894198141bfb955302402cfebec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dae07894198141bfb955302402cfebec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dae07894198141bfb955302402cfebec" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 43587 } } }
I20260806 02:52:30.429982 970 sys_catalog.cc:458] T 00000000000000000000000000000000 P dae07894198141bfb955302402cfebec [sys.catalog]: This master's current role is: LEADER
I20260806 02:52:30.435779 983 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:52:30.441105 983 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:52:30.450999 983 catalog_manager.cc:1347] Generated new cluster ID: 9591d9d98254483d9d3cfec8519e14a2
I20260806 02:52:30.451148 983 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:52:30.467442 983 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:52:30.468009 983 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:52:30.484066 983 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dae07894198141bfb955302402cfebec: Generated new TSK 0
I20260806 02:52:30.485163 983 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:52:30.496066 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.129:0
--local_ip_for_outbound_sockets=127.25.211.129
--webserver_interface=127.25.211.129
--webserver_port=0
--tserver_master_addrs=127.25.211.190:43587
--builtin_ntp_servers=127.25.211.148:33715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260806 02:52:30.855567 994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:52:30.855919 994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:52:30.856014 994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:52:30.863801 994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:52:30.863931 994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:52:30.864050 994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.129
I20260806 02:52:30.873797 994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:33715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.211.129
--webserver_port=0
--tserver_master_addrs=127.25.211.190:43587
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.129
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:52:30.875221 994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:52:30.877015 994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:52:30.890403 1000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:30.890467 1001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:30.890447 1006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:52:30.891417 994 server_base.cc:1034] running on GCE node
I20260806 02:52:30.892110 994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:52:30.893240 994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:52:30.894467 994 hybrid_clock.cc:648] HybridClock initialized: now 1785984750894408 us; error 51 us; skew 500 ppm
I20260806 02:52:30.894833 994 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:52:30.897343 994 webserver.cc:460] Webserver started at http://127.25.211.129:39707/ using document root <none> and password file <none>
I20260806 02:52:30.898098 994 fs_manager.cc:359] Metadata directory not provided
I20260806 02:52:30.898233 994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:52:30.898578 994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:52:30.901643 994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/ts-0/data/instance:
uuid: "7af661bde6c24df0a4cbd3047977edf2"
format_stamp: "Formatted at 2026-08-06 02:52:30 on dist-test-slave-zj8q"
server_key: "a0c2800b50838ac4129f55b8e511381e"
server_key_iv: "c431bc9023e450f8937fc7cdfab6aab1"
server_key_version: "encryptionkey@0"
I20260806 02:52:30.902571 994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/ts-0/wal/instance:
uuid: "7af661bde6c24df0a4cbd3047977edf2"
format_stamp: "Formatted at 2026-08-06 02:52:30 on dist-test-slave-zj8q"
server_key: "a0c2800b50838ac4129f55b8e511381e"
server_key_iv: "c431bc9023e450f8937fc7cdfab6aab1"
server_key_version: "encryptionkey@0"
I20260806 02:52:30.908586 994 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.002s
I20260806 02:52:30.912729 1014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:30.914273 994 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260806 02:52:30.914450 994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/ts-0/wal
uuid: "7af661bde6c24df0a4cbd3047977edf2"
format_stamp: "Formatted at 2026-08-06 02:52:30 on dist-test-slave-zj8q"
server_key: "a0c2800b50838ac4129f55b8e511381e"
server_key_iv: "c431bc9023e450f8937fc7cdfab6aab1"
server_key_version: "encryptionkey@0"
I20260806 02:52:30.914685 994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:52:30.920611 994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:52:30.921523 994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:52:30.921798 994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:52:30.922865 994 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:52:30.924403 994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:52:30.924505 994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:30.924619 994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:52:30.924697 994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:30.955516 994 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.129:41601
I20260806 02:52:30.955571 1179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.129:41601 every 8 connection(s)
I20260806 02:52:30.956888 994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
I20260806 02:52:30.965658 1184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:30.959635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:47349 (local address 127.25.211.190:43587)
0806 02:52:30.960100 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:30.960105 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:30.961298 (+ 1193us) server_negotiation.cc:407] Connection header received
0806 02:52:30.963187 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:30.963193 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:30.963302 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:30.963418 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:30.964853 (+ 1435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:30.964858 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:30.964861 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:52:30.964879 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:30.964927 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:52:30.964939 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:30.965095 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:30.965346 (+ 251us) server_negotiation.cc:299] Negotiation successful
0806 02:52:30.965403 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":256,"threads_started":1}
I20260806 02:52:30.966491 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 994
I20260806 02:52:30.966675 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/ts-0/wal/instance
I20260806 02:52:30.966722 1185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:30.960176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:43587 (local address 127.25.211.129:47349)
0806 02:52:30.961114 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:30.961152 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:30.962861 (+ 1709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:30.963749 (+ 888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:30.963758 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:30.963861 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:30.964530 (+ 669us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:52:30.964532 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:30.964583 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:52:30.964681 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:30.965115 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:30.965125 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:30.965132 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 02:52:30.965402 (+ 270us) client_negotiation.cc:240] Negotiation successful
0806 02:52:30.965652 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":277,"threads_started":1}
I20260806 02:52:30.967492 26446 external_mini_cluster.cc:1428] Setting key 8ae8aa217aa9a0ee38b57f92cf3b1234
I20260806 02:52:30.968429 1180 heartbeater.cc:347] Connected to a master server at 127.25.211.190:43587
I20260806 02:52:30.968783 1180 heartbeater.cc:464] Registering TS with master...
I20260806 02:52:30.969652 1180 heartbeater.cc:511] Master 127.25.211.190:43587 requested a full tablet report, sending...
I20260806 02:52:30.970707 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.130:0
--local_ip_for_outbound_sockets=127.25.211.130
--webserver_interface=127.25.211.130
--webserver_port=0
--tserver_master_addrs=127.25.211.190:43587
--builtin_ntp_servers=127.25.211.148:33715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260806 02:52:30.971853 917 ts_manager.cc:194] Registered new tserver with Master: 7af661bde6c24df0a4cbd3047977edf2 (127.25.211.129:41601)
W20260806 02:52:31.310088 1186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:52:31.310431 1186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:52:31.310525 1186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:52:31.317977 1186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:52:31.318099 1186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:52:31.318241 1186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.130
I20260806 02:52:31.327822 1186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:33715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.211.130
--webserver_port=0
--tserver_master_addrs=127.25.211.190:43587
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.130
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:52:31.329200 1186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:52:31.331127 1186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:52:31.344790 1192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:31.344885 1193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:31.344908 1195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:52:31.346014 1186 server_base.cc:1034] running on GCE node
I20260806 02:52:31.346705 1186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:52:31.347762 1186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:52:31.348976 1186 hybrid_clock.cc:648] HybridClock initialized: now 1785984751348929 us; error 37 us; skew 500 ppm
I20260806 02:52:31.349290 1186 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:52:31.351756 1186 webserver.cc:460] Webserver started at http://127.25.211.130:34095/ using document root <none> and password file <none>
I20260806 02:52:31.352478 1186 fs_manager.cc:359] Metadata directory not provided
I20260806 02:52:31.352576 1186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:52:31.352890 1186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:52:31.356320 1186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/ts-1/data/instance:
uuid: "3e6c1995bb2b451fbd53de7032f0f365"
format_stamp: "Formatted at 2026-08-06 02:52:31 on dist-test-slave-zj8q"
server_key: "79747d4643deb75015ed314526a75564"
server_key_iv: "7e62143f52fee59b8acdeceb9d909785"
server_key_version: "encryptionkey@0"
I20260806 02:52:31.357161 1186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/ts-1/wal/instance:
uuid: "3e6c1995bb2b451fbd53de7032f0f365"
format_stamp: "Formatted at 2026-08-06 02:52:31 on dist-test-slave-zj8q"
server_key: "79747d4643deb75015ed314526a75564"
server_key_iv: "7e62143f52fee59b8acdeceb9d909785"
server_key_version: "encryptionkey@0"
I20260806 02:52:31.362923 1186 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260806 02:52:31.366773 1206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:31.368188 1186 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:52:31.368353 1186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/ts-1/wal
uuid: "3e6c1995bb2b451fbd53de7032f0f365"
format_stamp: "Formatted at 2026-08-06 02:52:31 on dist-test-slave-zj8q"
server_key: "79747d4643deb75015ed314526a75564"
server_key_iv: "7e62143f52fee59b8acdeceb9d909785"
server_key_version: "encryptionkey@0"
I20260806 02:52:31.368561 1186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:52:31.374599 1186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:52:31.375473 1186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:52:31.375748 1186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:52:31.376792 1186 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:52:31.378353 1186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:52:31.378443 1186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:31.378554 1186 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:52:31.378630 1186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:31.412232 1186 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.130:42831
I20260806 02:52:31.412318 1366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.130:42831 every 8 connection(s)
I20260806 02:52:31.413684 1186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
I20260806 02:52:31.415395 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 1186
I20260806 02:52:31.415558 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/ts-1/wal/instance
I20260806 02:52:31.416090 26446 external_mini_cluster.cc:1428] Setting key 535e576c69f49d7a3fc71b6f0c8d7f4e
I20260806 02:52:31.419140 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.131:0
--local_ip_for_outbound_sockets=127.25.211.131
--webserver_interface=127.25.211.131
--webserver_port=0
--tserver_master_addrs=127.25.211.190:43587
--builtin_ntp_servers=127.25.211.148:33715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260806 02:52:31.422731 1184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:31.416409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:60899 (local address 127.25.211.190:43587)
0806 02:52:31.416634 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:31.416641 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:31.418702 (+ 2061us) server_negotiation.cc:407] Connection header received
0806 02:52:31.420493 (+ 1791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:31.420498 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:31.420568 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:31.420654 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:31.421972 (+ 1318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:31.421978 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:31.421983 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:52:31.422001 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:31.422037 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:52:31.422049 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:31.422176 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:31.422375 (+ 199us) server_negotiation.cc:299] Negotiation successful
0806 02:52:31.422445 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 02:52:31.423717 1372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:31.416927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:43587 (local address 127.25.211.130:60899)
0806 02:52:31.418549 (+ 1622us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:31.418596 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:31.420214 (+ 1618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:31.420914 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:31.420923 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:31.421017 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:31.421656 (+ 639us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:52:31.421659 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:31.421698 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:52:31.421799 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:31.422138 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:31.422169 (+ 31us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:31.422179 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 02:52:31.422450 (+ 271us) client_negotiation.cc:240] Negotiation successful
0806 02:52:31.422741 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1053,"thread_start_us":803,"threads_started":1}
I20260806 02:52:31.425210 1370 heartbeater.cc:347] Connected to a master server at 127.25.211.190:43587
I20260806 02:52:31.425542 1370 heartbeater.cc:464] Registering TS with master...
I20260806 02:52:31.426203 1370 heartbeater.cc:511] Master 127.25.211.190:43587 requested a full tablet report, sending...
I20260806 02:52:31.427850 917 ts_manager.cc:194] Registered new tserver with Master: 3e6c1995bb2b451fbd53de7032f0f365 (127.25.211.130:42831)
W20260806 02:52:31.765691 1375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:52:31.766019 1375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:52:31.766181 1375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:52:31.773816 1375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:52:31.773944 1375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:52:31.774101 1375 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.131
I20260806 02:52:31.783607 1375 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:33715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.211.131
--webserver_port=0
--tserver_master_addrs=127.25.211.190:43587
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.131
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:52:31.784966 1375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:52:31.786939 1375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:52:31.800799 1381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:31.800832 1382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:31.801141 1384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:52:31.801625 1375 server_base.cc:1034] running on GCE node
I20260806 02:52:31.802371 1375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:52:31.803404 1375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:52:31.804622 1375 hybrid_clock.cc:648] HybridClock initialized: now 1785984751804554 us; error 60 us; skew 500 ppm
I20260806 02:52:31.804946 1375 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:52:31.807485 1375 webserver.cc:460] Webserver started at http://127.25.211.131:42545/ using document root <none> and password file <none>
I20260806 02:52:31.808216 1375 fs_manager.cc:359] Metadata directory not provided
I20260806 02:52:31.808336 1375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:52:31.808677 1375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:52:31.811803 1375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/ts-2/data/instance:
uuid: "21a81af0b9704c069f9723cc1aeeb678"
format_stamp: "Formatted at 2026-08-06 02:52:31 on dist-test-slave-zj8q"
server_key: "271d7327163edd33f1cadeb7f1e71714"
server_key_iv: "5037f14b6e4786664a7ba66ba0271ad4"
server_key_version: "encryptionkey@0"
I20260806 02:52:31.812676 1375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/ts-2/wal/instance:
uuid: "21a81af0b9704c069f9723cc1aeeb678"
format_stamp: "Formatted at 2026-08-06 02:52:31 on dist-test-slave-zj8q"
server_key: "271d7327163edd33f1cadeb7f1e71714"
server_key_iv: "5037f14b6e4786664a7ba66ba0271ad4"
server_key_version: "encryptionkey@0"
I20260806 02:52:31.818449 1375 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260806 02:52:31.822207 1395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:31.823642 1375 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:52:31.823837 1375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/ts-2/wal
uuid: "21a81af0b9704c069f9723cc1aeeb678"
format_stamp: "Formatted at 2026-08-06 02:52:31 on dist-test-slave-zj8q"
server_key: "271d7327163edd33f1cadeb7f1e71714"
server_key_iv: "5037f14b6e4786664a7ba66ba0271ad4"
server_key_version: "encryptionkey@0"
I20260806 02:52:31.824083 1375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:52:31.829952 1375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:52:31.830879 1375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:52:31.831157 1375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:52:31.832235 1375 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:52:31.833925 1375 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:52:31.834043 1375 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:31.834190 1375 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:52:31.834271 1375 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260806 02:52:31.866886 1375 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.131:40227
I20260806 02:52:31.866951 1558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.131:40227 every 8 connection(s)
I20260806 02:52:31.868341 1375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
I20260806 02:52:31.875455 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 1375
I20260806 02:52:31.875641 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/ts-2/wal/instance
I20260806 02:52:31.876121 26446 external_mini_cluster.cc:1428] Setting key 0d37590d3c14f719dbe0f49ddbcd3d3e
I20260806 02:52:31.877331 1184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:31.871133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:35389 (local address 127.25.211.190:43587)
0806 02:52:31.871322 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:31.871327 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:31.873016 (+ 1689us) server_negotiation.cc:407] Connection header received
0806 02:52:31.874945 (+ 1929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:31.874952 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:31.875033 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:31.875164 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:31.876558 (+ 1394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:31.876563 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:31.876568 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:52:31.876590 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:31.876632 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:52:31.876646 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:31.876749 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:31.877015 (+ 266us) server_negotiation.cc:299] Negotiation successful
0806 02:52:31.877078 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 02:52:31.878624 1561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:31.871699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:43587 (local address 127.25.211.131:35389)
0806 02:52:31.872846 (+ 1147us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:31.872894 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:31.874660 (+ 1766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:31.875384 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:31.875393 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:31.875493 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:31.876199 (+ 706us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:52:31.876202 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:31.876268 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:52:31.876376 (+ 108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:31.876770 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:31.876783 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:31.876792 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 02:52:31.877079 (+ 287us) client_negotiation.cc:240] Negotiation successful
0806 02:52:31.877390 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":385,"threads_started":1}
I20260806 02:52:31.880043 1559 heartbeater.cc:347] Connected to a master server at 127.25.211.190:43587
I20260806 02:52:31.880374 1559 heartbeater.cc:464] Registering TS with master...
I20260806 02:52:31.880963 1559 heartbeater.cc:511] Master 127.25.211.190:43587 requested a full tablet report, sending...
I20260806 02:52:31.882431 917 ts_manager.cc:194] Registered new tserver with Master: 21a81af0b9704c069f9723cc1aeeb678 (127.25.211.131:40227)
I20260806 02:52:31.892154 26446 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:52:31.899137 1184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:31.896610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57368 (local address 127.25.211.190:43587)
0806 02:52:31.896803 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:31.896810 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:31.897320 (+ 510us) server_negotiation.cc:407] Connection header received
0806 02:52:31.897500 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:31.897506 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:31.897568 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:31.897653 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:31.898581 (+ 928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:31.898586 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:31.898589 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:52:31.898606 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:31.898641 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:52:31.898652 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:31.898772 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:31.898816 (+ 44us) server_negotiation.cc:299] Negotiation successful
0806 02:52:31.898866 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 02:52:31.903390 917 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57368:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:52:31.910363 917 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:52:31.940816 1577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:31.924362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.131:40227 (local address 127.0.0.1:39598)
0806 02:52:31.937579 (+ 13217us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:31.937596 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:31.937697 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:31.938439 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:31.938443 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:31.938466 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:31.939343 (+ 877us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:52:31.939347 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:31.939363 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:52:31.939405 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:31.940133 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:31.940137 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:31.940142 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 02:52:31.940295 (+ 153us) client_negotiation.cc:240] Negotiation successful
0806 02:52:31.940355 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13106,"thread_start_us":9413,"threads_started":1}
I20260806 02:52:31.940929 1576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:31.922918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34496 (local address 127.25.211.130:42831)
0806 02:52:31.936082 (+ 13164us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:31.936093 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:31.936130 (+ 37us) server_negotiation.cc:407] Connection header received
0806 02:52:31.936210 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:31.936218 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:31.936507 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:31.936704 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:31.939581 (+ 2877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:31.939603 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:31.939620 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:52:31.939641 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:31.939717 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:52:31.939740 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:31.939896 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:31.940507 (+ 611us) server_negotiation.cc:299] Negotiation successful
0806 02:52:31.940640 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13004,"thread_start_us":12879,"threads_started":1}
I20260806 02:52:31.941263 1581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:31.924626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39598 (local address 127.25.211.131:40227)
0806 02:52:31.936547 (+ 11921us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:31.936555 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:31.937814 (+ 1259us) server_negotiation.cc:407] Connection header received
0806 02:52:31.937890 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:31.937897 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:31.938132 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:31.938316 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:31.939901 (+ 1585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:31.939908 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:31.939921 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:52:31.939938 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:31.939996 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:52:31.940015 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:31.940193 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:31.940519 (+ 326us) server_negotiation.cc:299] Negotiation successful
0806 02:52:31.940622 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11750,"thread_start_us":11605,"threads_started":1}
I20260806 02:52:31.941452 1580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:31.922838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.130:42831 (local address 127.0.0.1:34496)
0806 02:52:31.934290 (+ 11452us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:31.934345 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:31.934510 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:31.937781 (+ 3271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:31.937789 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:31.937894 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:31.938992 (+ 1098us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:52:31.938997 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:31.939031 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:52:31.939118 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:31.939960 (+ 842us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:31.939977 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:31.939985 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 02:52:31.940191 (+ 206us) client_negotiation.cc:240] Negotiation successful
0806 02:52:31.940296 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11134,"spinlock_wait_cycles":25984,"thread_start_us":13865,"threads_started":1}
I20260806 02:52:31.942509 1578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:31.922897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:41601 (local address 127.0.0.1:33026)
0806 02:52:31.936996 (+ 14099us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:31.937017 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:31.937133 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:31.939855 (+ 2722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:31.939860 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:31.939884 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:31.940559 (+ 675us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:52:31.940563 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:31.940578 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:52:31.940627 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:31.941877 (+ 1250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:31.941882 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:31.941887 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 02:52:31.941999 (+ 112us) client_negotiation.cc:240] Negotiation successful
0806 02:52:31.942052 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13947,"thread_start_us":14440,"threads_started":1}
I20260806 02:52:31.943717 1579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:31.923431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33026 (local address 127.25.211.129:41601)
0806 02:52:31.939100 (+ 15669us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:31.939119 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:31.939150 (+ 31us) server_negotiation.cc:407] Connection header received
0806 02:52:31.939223 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:31.939231 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:31.939428 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:31.940318 (+ 890us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:31.941609 (+ 1291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:31.941617 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:31.941630 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:52:31.941657 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:31.941714 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:52:31.941739 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:31.943089 (+ 1350us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:31.943172 (+ 83us) server_negotiation.cc:299] Negotiation successful
0806 02:52:31.943273 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15495,"thread_start_us":16126,"threads_started":1}
I20260806 02:52:31.945604 1463 tablet_service.cc:1467] Processing CreateTablet for tablet 3bc4cb4fdc0e465a96459601fdbadec1 (DEFAULT_TABLE table=test-table [id=4c9162218d27447497914358a9f25397]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:52:31.945609 1281 tablet_service.cc:1467] Processing CreateTablet for tablet 3bc4cb4fdc0e465a96459601fdbadec1 (DEFAULT_TABLE table=test-table [id=4c9162218d27447497914358a9f25397]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:52:31.946769 1084 tablet_service.cc:1467] Processing CreateTablet for tablet 3bc4cb4fdc0e465a96459601fdbadec1 (DEFAULT_TABLE table=test-table [id=4c9162218d27447497914358a9f25397]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:52:31.947166 1281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bc4cb4fdc0e465a96459601fdbadec1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:31.947166 1463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bc4cb4fdc0e465a96459601fdbadec1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:31.947815 1084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bc4cb4fdc0e465a96459601fdbadec1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:31.967747 1583 tablet_bootstrap.cc:492] T 3bc4cb4fdc0e465a96459601fdbadec1 P 7af661bde6c24df0a4cbd3047977edf2: Bootstrap starting.
I20260806 02:52:31.968067 1585 tablet_bootstrap.cc:492] T 3bc4cb4fdc0e465a96459601fdbadec1 P 3e6c1995bb2b451fbd53de7032f0f365: Bootstrap starting.
I20260806 02:52:31.971948 1583 tablet_bootstrap.cc:654] T 3bc4cb4fdc0e465a96459601fdbadec1 P 7af661bde6c24df0a4cbd3047977edf2: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:31.973073 1586 tablet_bootstrap.cc:492] T 3bc4cb4fdc0e465a96459601fdbadec1 P 21a81af0b9704c069f9723cc1aeeb678: Bootstrap starting.
I20260806 02:52:31.973672 1583 log.cc:824] T 3bc4cb4fdc0e465a96459601fdbadec1 P 7af661bde6c24df0a4cbd3047977edf2: Log is configured to *not* fsync() on all Append() calls
I20260806 02:52:31.974411 1585 tablet_bootstrap.cc:654] T 3bc4cb4fdc0e465a96459601fdbadec1 P 3e6c1995bb2b451fbd53de7032f0f365: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:31.975003 1180 heartbeater.cc:503] Master 127.25.211.190:43587 was elected leader, sending a full tablet report...
I20260806 02:52:31.975718 1585 log.cc:824] T 3bc4cb4fdc0e465a96459601fdbadec1 P 3e6c1995bb2b451fbd53de7032f0f365: Log is configured to *not* fsync() on all Append() calls
I20260806 02:52:31.976985 1586 tablet_bootstrap.cc:654] T 3bc4cb4fdc0e465a96459601fdbadec1 P 21a81af0b9704c069f9723cc1aeeb678: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:31.978008 1586 log.cc:824] T 3bc4cb4fdc0e465a96459601fdbadec1 P 21a81af0b9704c069f9723cc1aeeb678: Log is configured to *not* fsync() on all Append() calls
I20260806 02:52:31.978570 1583 tablet_bootstrap.cc:492] T 3bc4cb4fdc0e465a96459601fdbadec1 P 7af661bde6c24df0a4cbd3047977edf2: No bootstrap required, opened a new log
I20260806 02:52:31.978909 1583 ts_tablet_manager.cc:1397] T 3bc4cb4fdc0e465a96459601fdbadec1 P 7af661bde6c24df0a4cbd3047977edf2: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20260806 02:52:31.980486 1585 tablet_bootstrap.cc:492] T 3bc4cb4fdc0e465a96459601fdbadec1 P 3e6c1995bb2b451fbd53de7032f0f365: No bootstrap required, opened a new log
I20260806 02:52:31.980758 1585 ts_tablet_manager.cc:1397] T 3bc4cb4fdc0e465a96459601fdbadec1 P 3e6c1995bb2b451fbd53de7032f0f365: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.003s
I20260806 02:52:31.985034 1586 tablet_bootstrap.cc:492] T 3bc4cb4fdc0e465a96459601fdbadec1 P 21a81af0b9704c069f9723cc1aeeb678: No bootstrap required, opened a new log
I20260806 02:52:31.985363 1586 ts_tablet_manager.cc:1397] T 3bc4cb4fdc0e465a96459601fdbadec1 P 21a81af0b9704c069f9723cc1aeeb678: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20260806 02:52:31.988343 1585 raft_consensus.cc:348] T 3bc4cb4fdc0e465a96459601fdbadec1 P 3e6c1995bb2b451fbd53de7032f0f365 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6c1995bb2b451fbd53de7032f0f365" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 42831 } } peers { permanent_uuid: "7af661bde6c24df0a4cbd3047977edf2" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 41601 } } peers { permanent_uuid: "21a81af0b9704c069f9723cc1aeeb678" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 40227 } }
I20260806 02:52:31.988770 1585 raft_consensus.cc:374] T 3bc4cb4fdc0e465a96459601fdbadec1 P 3e6c1995bb2b451fbd53de7032f0f365 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:31.988890 1585 raft_consensus.cc:729] T 3bc4cb4fdc0e465a96459601fdbadec1 P 3e6c1995bb2b451fbd53de7032f0f365 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e6c1995bb2b451fbd53de7032f0f365, State: Initialized, Role: FOLLOWER
I20260806 02:52:31.988673 1583 raft_consensus.cc:348] T 3bc4cb4fdc0e465a96459601fdbadec1 P 7af661bde6c24df0a4cbd3047977edf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6c1995bb2b451fbd53de7032f0f365" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 42831 } } peers { permanent_uuid: "7af661bde6c24df0a4cbd3047977edf2" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 41601 } } peers { permanent_uuid: "21a81af0b9704c069f9723cc1aeeb678" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 40227 } }
I20260806 02:52:31.989171 1583 raft_consensus.cc:374] T 3bc4cb4fdc0e465a96459601fdbadec1 P 7af661bde6c24df0a4cbd3047977edf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:31.989303 1583 raft_consensus.cc:729] T 3bc4cb4fdc0e465a96459601fdbadec1 P 7af661bde6c24df0a4cbd3047977edf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7af661bde6c24df0a4cbd3047977edf2, State: Initialized, Role: FOLLOWER
I20260806 02:52:31.989574 1585 consensus_queue.cc:260] T 3bc4cb4fdc0e465a96459601fdbadec1 P 3e6c1995bb2b451fbd53de7032f0f365 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6c1995bb2b451fbd53de7032f0f365" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 42831 } } peers { permanent_uuid: "7af661bde6c24df0a4cbd3047977edf2" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 41601 } } peers { permanent_uuid: "21a81af0b9704c069f9723cc1aeeb678" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 40227 } }
I20260806 02:52:31.989912 1583 consensus_queue.cc:260] T 3bc4cb4fdc0e465a96459601fdbadec1 P 7af661bde6c24df0a4cbd3047977edf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6c1995bb2b451fbd53de7032f0f365" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 42831 } } peers { permanent_uuid: "7af661bde6c24df0a4cbd3047977edf2" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 41601 } } peers { permanent_uuid: "21a81af0b9704c069f9723cc1aeeb678" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 40227 } }
I20260806 02:52:31.990830 1370 heartbeater.cc:503] Master 127.25.211.190:43587 was elected leader, sending a full tablet report...
I20260806 02:52:31.991385 1585 ts_tablet_manager.cc:1428] T 3bc4cb4fdc0e465a96459601fdbadec1 P 3e6c1995bb2b451fbd53de7032f0f365: Time spent starting tablet: real 0.010s user 0.009s sys 0.002s
I20260806 02:52:31.993479 1583 ts_tablet_manager.cc:1428] T 3bc4cb4fdc0e465a96459601fdbadec1 P 7af661bde6c24df0a4cbd3047977edf2: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260806 02:52:31.998416 1586 raft_consensus.cc:348] T 3bc4cb4fdc0e465a96459601fdbadec1 P 21a81af0b9704c069f9723cc1aeeb678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6c1995bb2b451fbd53de7032f0f365" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 42831 } } peers { permanent_uuid: "7af661bde6c24df0a4cbd3047977edf2" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 41601 } } peers { permanent_uuid: "21a81af0b9704c069f9723cc1aeeb678" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 40227 } }
I20260806 02:52:31.998816 1586 raft_consensus.cc:374] T 3bc4cb4fdc0e465a96459601fdbadec1 P 21a81af0b9704c069f9723cc1aeeb678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:31.998893 1586 raft_consensus.cc:729] T 3bc4cb4fdc0e465a96459601fdbadec1 P 21a81af0b9704c069f9723cc1aeeb678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21a81af0b9704c069f9723cc1aeeb678, State: Initialized, Role: FOLLOWER
I20260806 02:52:31.999441 1586 consensus_queue.cc:260] T 3bc4cb4fdc0e465a96459601fdbadec1 P 21a81af0b9704c069f9723cc1aeeb678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6c1995bb2b451fbd53de7032f0f365" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 42831 } } peers { permanent_uuid: "7af661bde6c24df0a4cbd3047977edf2" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 41601 } } peers { permanent_uuid: "21a81af0b9704c069f9723cc1aeeb678" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 40227 } }
I20260806 02:52:32.000515 1559 heartbeater.cc:503] Master 127.25.211.190:43587 was elected leader, sending a full tablet report...
I20260806 02:52:32.001210 1586 ts_tablet_manager.cc:1428] T 3bc4cb4fdc0e465a96459601fdbadec1 P 21a81af0b9704c069f9723cc1aeeb678: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260806 02:52:32.038470 1591 raft_consensus.cc:482] T 3bc4cb4fdc0e465a96459601fdbadec1 P 7af661bde6c24df0a4cbd3047977edf2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:52:32.038877 1591 raft_consensus.cc:504] T 3bc4cb4fdc0e465a96459601fdbadec1 P 7af661bde6c24df0a4cbd3047977edf2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6c1995bb2b451fbd53de7032f0f365" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 42831 } } peers { permanent_uuid: "7af661bde6c24df0a4cbd3047977edf2" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 41601 } } peers { permanent_uuid: "21a81af0b9704c069f9723cc1aeeb678" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 40227 } }
I20260806 02:52:32.040683 1591 leader_election.cc:288] T 3bc4cb4fdc0e465a96459601fdbadec1 P 7af661bde6c24df0a4cbd3047977edf2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e6c1995bb2b451fbd53de7032f0f365 (127.25.211.130:42831), 21a81af0b9704c069f9723cc1aeeb678 (127.25.211.131:40227)
I20260806 02:52:32.044521 1576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:32.041180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:55549 (local address 127.25.211.130:42831)
0806 02:52:32.041381 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:32.041388 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:32.042146 (+ 758us) server_negotiation.cc:407] Connection header received
0806 02:52:32.042367 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:32.042373 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:32.042428 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:32.042534 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:32.043853 (+ 1319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:32.043859 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:32.043864 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:52:32.043889 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:32.043929 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:52:32.043945 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:32.044069 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:32.044178 (+ 109us) server_negotiation.cc:299] Negotiation successful
0806 02:52:32.044246 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260806 02:52:32.044514 1581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:32.041274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:51647 (local address 127.25.211.131:40227)
0806 02:52:32.041435 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:32.041441 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:32.042147 (+ 706us) server_negotiation.cc:407] Connection header received
0806 02:52:32.042343 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:32.042349 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:32.042420 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:32.042537 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:32.043853 (+ 1316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:32.043859 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:32.043864 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:52:32.043889 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:32.043929 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:52:32.043945 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:32.044073 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:32.044178 (+ 105us) server_negotiation.cc:299] Negotiation successful
0806 02:52:32.044245 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 02:52:32.044903 1594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:32.041101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.131:40227 (local address 127.25.211.129:51647)
0806 02:52:32.041997 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:32.042023 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:32.042207 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:32.042580 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:32.042585 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:32.042613 (+ 28us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:32.043506 (+ 893us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:52:32.043509 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:32.043531 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:52:32.043610 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:32.044060 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:32.044065 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:32.044070 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 02:52:32.044245 (+ 175us) client_negotiation.cc:240] Negotiation successful
0806 02:52:32.044311 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"mutex_wait_us":21,"thread_start_us":446,"threads_started":1}
I20260806 02:52:32.044929 1593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:32.041121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.130:42831 (local address 127.25.211.129:55549)
0806 02:52:32.042034 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:32.042049 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:32.042261 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:32.042580 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:32.042585 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:32.042613 (+ 28us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:32.043545 (+ 932us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:52:32.043548 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:32.043559 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:52:32.043613 (+ 54us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:32.044060 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:32.044065 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:32.044070 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 02:52:32.044227 (+ 157us) client_negotiation.cc:240] Negotiation successful
0806 02:52:32.044297 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":815,"mutex_wait_us":53,"spinlock_wait_cycles":27008,"thread_start_us":407,"threads_started":1}
I20260806 02:52:32.046132 1490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bc4cb4fdc0e465a96459601fdbadec1" candidate_uuid: "7af661bde6c24df0a4cbd3047977edf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21a81af0b9704c069f9723cc1aeeb678" is_pre_election: true
I20260806 02:52:32.046136 1303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bc4cb4fdc0e465a96459601fdbadec1" candidate_uuid: "7af661bde6c24df0a4cbd3047977edf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e6c1995bb2b451fbd53de7032f0f365" is_pre_election: true
I20260806 02:52:32.046803 1490 raft_consensus.cc:2454] T 3bc4cb4fdc0e465a96459601fdbadec1 P 21a81af0b9704c069f9723cc1aeeb678 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7af661bde6c24df0a4cbd3047977edf2 in term 0.
I20260806 02:52:32.046797 1303 raft_consensus.cc:2454] T 3bc4cb4fdc0e465a96459601fdbadec1 P 3e6c1995bb2b451fbd53de7032f0f365 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7af661bde6c24df0a4cbd3047977edf2 in term 0.
I20260806 02:52:32.047586 1016 leader_election.cc:302] T 3bc4cb4fdc0e465a96459601fdbadec1 P 7af661bde6c24df0a4cbd3047977edf2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 21a81af0b9704c069f9723cc1aeeb678, 3e6c1995bb2b451fbd53de7032f0f365, 7af661bde6c24df0a4cbd3047977edf2; no voters:
I20260806 02:52:32.048096 1591 raft_consensus.cc:2781] T 3bc4cb4fdc0e465a96459601fdbadec1 P 7af661bde6c24df0a4cbd3047977edf2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:52:32.048254 1591 raft_consensus.cc:482] T 3bc4cb4fdc0e465a96459601fdbadec1 P 7af661bde6c24df0a4cbd3047977edf2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:52:32.048350 1591 raft_consensus.cc:3037] T 3bc4cb4fdc0e465a96459601fdbadec1 P 7af661bde6c24df0a4cbd3047977edf2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:32.050650 1591 raft_consensus.cc:504] T 3bc4cb4fdc0e465a96459601fdbadec1 P 7af661bde6c24df0a4cbd3047977edf2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6c1995bb2b451fbd53de7032f0f365" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 42831 } } peers { permanent_uuid: "7af661bde6c24df0a4cbd3047977edf2" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 41601 } } peers { permanent_uuid: "21a81af0b9704c069f9723cc1aeeb678" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 40227 } }
I20260806 02:52:32.051259 1591 leader_election.cc:288] T 3bc4cb4fdc0e465a96459601fdbadec1 P 7af661bde6c24df0a4cbd3047977edf2 [CANDIDATE]: Term 1 election: Requested vote from peers 3e6c1995bb2b451fbd53de7032f0f365 (127.25.211.130:42831), 21a81af0b9704c069f9723cc1aeeb678 (127.25.211.131:40227)
I20260806 02:52:32.051863 1490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bc4cb4fdc0e465a96459601fdbadec1" candidate_uuid: "7af661bde6c24df0a4cbd3047977edf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21a81af0b9704c069f9723cc1aeeb678"
I20260806 02:52:32.051863 1303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bc4cb4fdc0e465a96459601fdbadec1" candidate_uuid: "7af661bde6c24df0a4cbd3047977edf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e6c1995bb2b451fbd53de7032f0f365"
I20260806 02:52:32.052110 1303 raft_consensus.cc:3037] T 3bc4cb4fdc0e465a96459601fdbadec1 P 3e6c1995bb2b451fbd53de7032f0f365 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:32.052098 1490 raft_consensus.cc:3037] T 3bc4cb4fdc0e465a96459601fdbadec1 P 21a81af0b9704c069f9723cc1aeeb678 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:32.055534 1490 raft_consensus.cc:2454] T 3bc4cb4fdc0e465a96459601fdbadec1 P 21a81af0b9704c069f9723cc1aeeb678 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7af661bde6c24df0a4cbd3047977edf2 in term 1.
I20260806 02:52:32.055523 1303 raft_consensus.cc:2454] T 3bc4cb4fdc0e465a96459601fdbadec1 P 3e6c1995bb2b451fbd53de7032f0f365 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7af661bde6c24df0a4cbd3047977edf2 in term 1.
I20260806 02:52:32.056195 1021 leader_election.cc:302] T 3bc4cb4fdc0e465a96459601fdbadec1 P 7af661bde6c24df0a4cbd3047977edf2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 21a81af0b9704c069f9723cc1aeeb678, 3e6c1995bb2b451fbd53de7032f0f365, 7af661bde6c24df0a4cbd3047977edf2; no voters:
I20260806 02:52:32.056576 1591 raft_consensus.cc:2781] T 3bc4cb4fdc0e465a96459601fdbadec1 P 7af661bde6c24df0a4cbd3047977edf2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:52:32.057096 1591 raft_consensus.cc:686] T 3bc4cb4fdc0e465a96459601fdbadec1 P 7af661bde6c24df0a4cbd3047977edf2 [term 1 LEADER]: Becoming Leader. State: Replica: 7af661bde6c24df0a4cbd3047977edf2, State: Running, Role: LEADER
I20260806 02:52:32.057688 1591 consensus_queue.cc:237] T 3bc4cb4fdc0e465a96459601fdbadec1 P 7af661bde6c24df0a4cbd3047977edf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6c1995bb2b451fbd53de7032f0f365" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 42831 } } peers { permanent_uuid: "7af661bde6c24df0a4cbd3047977edf2" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 41601 } } peers { permanent_uuid: "21a81af0b9704c069f9723cc1aeeb678" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 40227 } }
I20260806 02:52:32.062493 914 catalog_manager.cc:5495] T 3bc4cb4fdc0e465a96459601fdbadec1 P 7af661bde6c24df0a4cbd3047977edf2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7af661bde6c24df0a4cbd3047977edf2 (127.25.211.129). New cstate: current_term: 1 leader_uuid: "7af661bde6c24df0a4cbd3047977edf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e6c1995bb2b451fbd53de7032f0f365" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 42831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7af661bde6c24df0a4cbd3047977edf2" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 41601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21a81af0b9704c069f9723cc1aeeb678" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 40227 } health_report { overall_health: UNKNOWN } } }
I20260806 02:52:32.097347 1579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:32.095383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33030 (local address 127.25.211.129:41601)
0806 02:52:32.095561 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:32.095566 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:32.095585 (+ 19us) server_negotiation.cc:407] Connection header received
0806 02:52:32.095687 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:32.095691 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:32.095781 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:32.095873 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:32.096764 (+ 891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:32.096769 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:32.096772 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:52:32.096788 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:32.096823 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:52:32.096834 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:32.096929 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:32.096999 (+ 70us) server_negotiation.cc:299] Negotiation successful
0806 02:52:32.097055 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 02:52:32.106352 1303 raft_consensus.cc:1261] T 3bc4cb4fdc0e465a96459601fdbadec1 P 3e6c1995bb2b451fbd53de7032f0f365 [term 1 FOLLOWER]: Refusing update from remote peer 7af661bde6c24df0a4cbd3047977edf2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:52:32.106354 1490 raft_consensus.cc:1261] T 3bc4cb4fdc0e465a96459601fdbadec1 P 21a81af0b9704c069f9723cc1aeeb678 [term 1 FOLLOWER]: Refusing update from remote peer 7af661bde6c24df0a4cbd3047977edf2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:52:32.107497 1597 consensus_queue.cc:1035] T 3bc4cb4fdc0e465a96459601fdbadec1 P 7af661bde6c24df0a4cbd3047977edf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21a81af0b9704c069f9723cc1aeeb678" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 40227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:52:32.107890 1591 consensus_queue.cc:1035] T 3bc4cb4fdc0e465a96459601fdbadec1 P 7af661bde6c24df0a4cbd3047977edf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e6c1995bb2b451fbd53de7032f0f365" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 42831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:52:32.118829 1600 mvcc.cc:205] Tried to move back new op lower bound from 7315393544616738816 to 7315393544432676864. Current Snapshot: MvccSnapshot[applied={T|T < 7315393544616738816}]
W20260806 02:52:32.120059 1560 tablet.cc:2369] T 3bc4cb4fdc0e465a96459601fdbadec1 P 21a81af0b9704c069f9723cc1aeeb678: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:52:32.123833 1602 mvcc.cc:205] Tried to move back new op lower bound from 7315393544616738816 to 7315393544432676864. Current Snapshot: MvccSnapshot[applied={T|T < 7315393544616738816}]
I20260806 02:52:32.125416 1601 mvcc.cc:205] Tried to move back new op lower bound from 7315393544616738816 to 7315393544432676864. Current Snapshot: MvccSnapshot[applied={T|T < 7315393544616738816}]
I20260806 02:52:32.138289 914 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:52:32.138577 914 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:52:32.142134 914 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dae07894198141bfb955302402cfebec: Sending DeleteTablet for 3 replicas of tablet 3bc4cb4fdc0e465a96459601fdbadec1
I20260806 02:52:32.143344 1463 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bc4cb4fdc0e465a96459601fdbadec1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:52:32 UTC) from {username='slave'} at 127.0.0.1:39598
I20260806 02:52:32.143399 1084 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bc4cb4fdc0e465a96459601fdbadec1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:52:32 UTC) from {username='slave'} at 127.0.0.1:33026
I20260806 02:52:32.143399 1281 tablet_service.cc:1514] Processing DeleteTablet for tablet 3bc4cb4fdc0e465a96459601fdbadec1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:52:32 UTC) from {username='slave'} at 127.0.0.1:34496
I20260806 02:52:32.144503 1612 tablet_replica.cc:330] T 3bc4cb4fdc0e465a96459601fdbadec1 P 21a81af0b9704c069f9723cc1aeeb678: stopping tablet replica
I20260806 02:52:32.145071 1612 raft_consensus.cc:2229] T 3bc4cb4fdc0e465a96459601fdbadec1 P 21a81af0b9704c069f9723cc1aeeb678 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:52:32.145722 1612 raft_consensus.cc:2258] T 3bc4cb4fdc0e465a96459601fdbadec1 P 21a81af0b9704c069f9723cc1aeeb678 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:52:32.146415 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 994
I20260806 02:52:32.148301 1612 ts_tablet_manager.cc:1905] T 3bc4cb4fdc0e465a96459601fdbadec1 P 21a81af0b9704c069f9723cc1aeeb678: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:52:32.156165 1612 ts_tablet_manager.cc:1918] T 3bc4cb4fdc0e465a96459601fdbadec1 P 21a81af0b9704c069f9723cc1aeeb678: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:52:32.156513 1612 log.cc:1196] T 3bc4cb4fdc0e465a96459601fdbadec1 P 21a81af0b9704c069f9723cc1aeeb678: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/ts-2/wal/wals/3bc4cb4fdc0e465a96459601fdbadec1
I20260806 02:52:32.157179 1612 ts_tablet_manager.cc:1939] T 3bc4cb4fdc0e465a96459601fdbadec1 P 21a81af0b9704c069f9723cc1aeeb678: Deleting consensus metadata
I20260806 02:52:32.158890 1615 tablet_replica.cc:330] T 3bc4cb4fdc0e465a96459601fdbadec1 P 7af661bde6c24df0a4cbd3047977edf2: stopping tablet replica
I20260806 02:52:32.159332 892 catalog_manager.cc:4841] TS 21a81af0b9704c069f9723cc1aeeb678 (127.25.211.131:40227): tablet 3bc4cb4fdc0e465a96459601fdbadec1 (table test-table [id=4c9162218d27447497914358a9f25397]) successfully deleted
I20260806 02:52:32.160089 1615 raft_consensus.cc:2229] T 3bc4cb4fdc0e465a96459601fdbadec1 P 7af661bde6c24df0a4cbd3047977edf2 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:52:32.160885 1615 raft_consensus.cc:2258] T 3bc4cb4fdc0e465a96459601fdbadec1 P 7af661bde6c24df0a4cbd3047977edf2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:52:32.161186 1579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:32.149332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33044 (local address 127.25.211.129:41601)
0806 02:52:32.149787 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:32.149793 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:32.158316 (+ 8523us) server_negotiation.cc:407] Connection header received
0806 02:52:32.158869 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:32.158879 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:32.158946 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:32.159062 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:32.160433 (+ 1371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:32.160439 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:32.160444 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:52:32.160472 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:32.160514 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:52:32.160534 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:32.160671 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:32.160778 (+ 107us) server_negotiation.cc:299] Negotiation successful
0806 02:52:32.160842 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327}
I20260806 02:52:32.162828 1613 tablet_replica.cc:330] T 3bc4cb4fdc0e465a96459601fdbadec1 P 3e6c1995bb2b451fbd53de7032f0f365: stopping tablet replica
I20260806 02:52:32.163368 1613 raft_consensus.cc:2229] T 3bc4cb4fdc0e465a96459601fdbadec1 P 3e6c1995bb2b451fbd53de7032f0f365 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:52:32.163427 1175 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:52:32.163991 1613 raft_consensus.cc:2258] T 3bc4cb4fdc0e465a96459601fdbadec1 P 3e6c1995bb2b451fbd53de7032f0f365 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:52:32.166647 1613 ts_tablet_manager.cc:1905] T 3bc4cb4fdc0e465a96459601fdbadec1 P 3e6c1995bb2b451fbd53de7032f0f365: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:52:32.174364 1613 ts_tablet_manager.cc:1918] T 3bc4cb4fdc0e465a96459601fdbadec1 P 3e6c1995bb2b451fbd53de7032f0f365: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:52:32.174573 1613 log.cc:1196] T 3bc4cb4fdc0e465a96459601fdbadec1 P 3e6c1995bb2b451fbd53de7032f0f365: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1785984707115702-26446-0/minicluster-data/ts-1/wal/wals/3bc4cb4fdc0e465a96459601fdbadec1
I20260806 02:52:32.175117 1613 ts_tablet_manager.cc:1939] T 3bc4cb4fdc0e465a96459601fdbadec1 P 3e6c1995bb2b451fbd53de7032f0f365: Deleting consensus metadata
I20260806 02:52:32.177071 890 catalog_manager.cc:4841] TS 3e6c1995bb2b451fbd53de7032f0f365 (127.25.211.130:42831): tablet 3bc4cb4fdc0e465a96459601fdbadec1 (table test-table [id=4c9162218d27447497914358a9f25397]) successfully deleted
I20260806 02:52:32.260481 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 994
W20260806 02:52:32.283946 891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.211.129:41601 (error 108)
I20260806 02:52:32.284812 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 1186
I20260806 02:52:32.286070 1578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:32.285487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:41601 (local address 127.0.0.1:33060)
0806 02:52:32.285698 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:32.285892 (+ 194us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.129:41601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260806 02:52:32.286345 891 catalog_manager.cc:4584] TS 7af661bde6c24df0a4cbd3047977edf2 (127.25.211.129:41601): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3bc4cb4fdc0e465a96459601fdbadec1: Network error: Client connection negotiation failed: client connection to 127.25.211.129:41601: connect: Connection refused (error 111)
I20260806 02:52:32.287967 1576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:32.285658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34512 (local address 127.25.211.130:42831)
0806 02:52:32.285839 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:32.285845 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:32.285869 (+ 24us) server_negotiation.cc:407] Connection header received
0806 02:52:32.286015 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:32.286021 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:32.286139 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:32.286287 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:32.287199 (+ 912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:32.287206 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:32.287210 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:52:32.287236 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:32.287271 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:52:32.287284 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:32.287403 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:32.287518 (+ 115us) server_negotiation.cc:299] Negotiation successful
0806 02:52:32.287582 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 02:52:32.288714 1360 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:52:32.340086 1578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:32.339491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:41601 (local address 127.0.0.1:33076)
0806 02:52:32.339731 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:32.339856 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.129:41601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260806 02:52:32.341187 1578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:32.340776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:41601 (local address 127.0.0.1:33086)
0806 02:52:32.340933 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:32.341026 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.129:41601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260806 02:52:32.376860 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 1186
I20260806 02:52:32.401855 1578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:32.401283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:41601 (local address 127.0.0.1:33088)
0806 02:52:32.401508 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:32.401666 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.129:41601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260806 02:52:32.403084 1578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:32.402668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:41601 (local address 127.0.0.1:33094)
0806 02:52:32.402838 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:32.402933 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.129:41601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260806 02:52:32.406132 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 1375
I20260806 02:52:32.408752 1581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:32.406898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39614 (local address 127.25.211.131:40227)
0806 02:52:32.407096 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:32.407101 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:32.407120 (+ 19us) server_negotiation.cc:407] Connection header received
0806 02:52:32.407234 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:32.407238 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:32.407321 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:32.407407 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:32.408221 (+ 814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:32.408226 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:32.408229 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:52:32.408244 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:32.408276 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:52:32.408288 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:32.408392 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:32.408429 (+ 37us) server_negotiation.cc:299] Negotiation successful
0806 02:52:32.408474 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 02:52:32.409375 1552 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:52:32.484469 1578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:32.483916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:41601 (local address 127.0.0.1:33100)
0806 02:52:32.484202 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:32.484311 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.129:41601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
I20260806 02:52:32.485597 1578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:32.485173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:41601 (local address 127.0.0.1:33114)
0806 02:52:32.485384 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:32.485458 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.129:41601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260806 02:52:32.497380 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 1375
I20260806 02:52:32.519831 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 865
I20260806 02:52:32.521119 952 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:52:32.627606 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 865
2026-08-06T02:52:32Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2754 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260806 02:52:32.676033 26446 mini_oidc.cc:152] Starting JWKS server
I20260806 02:52:32.679262 26446 webserver.cc:460] Webserver started at https://127.0.0.1:44833/ using document root <none> and password file <none>
I20260806 02:52:32.679610 26446 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 02:52:32.680300 26446 webserver.cc:460] Webserver started at http://127.0.0.1:41767/ using document root <none> and password file <none>
2026-08-06T02:52:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:52:32Z Disabled control of system clock
I20260806 02:52:32.716850 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984707115702-26446-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984707115702-26446-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984707115702-26446-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:39709
--webserver_interface=127.25.211.190
--webserver_port=0
--builtin_ntp_servers=127.25.211.148:46067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.211.190:39709
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44833/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984707115702-26446-0/testchainca.cert with env {}
W20260806 02:52:33.058218 1632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:52:33.058568 1632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:52:33.058696 1632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:52:33.066121 1632 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:52:33.066296 1632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:52:33.066355 1632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:52:33.066434 1632 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:52:33.066493 1632 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 02:52:33.066583 1632 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 02:52:33.066639 1632 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44833/jwks.json
I20260806 02:52:33.077138 1632 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:46067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984707115702-26446-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984707115702-26446-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.211.190:39709
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:39709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44833/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.211.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984707115702-26446-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984707115702-26446-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:52:33.078593 1632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:52:33.079988 1632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:52:33.091957 1641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:33.092011 1642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:33.092161 1644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:52:33.092830 1632 server_base.cc:1034] running on GCE node
I20260806 02:52:33.093618 1632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:52:33.095134 1632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:52:33.096346 1632 hybrid_clock.cc:648] HybridClock initialized: now 1785984753096283 us; error 48 us; skew 500 ppm
I20260806 02:52:33.096705 1632 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:52:33.102065 1632 webserver.cc:460] Webserver started at http://127.25.211.190:46695/ using document root <none> and password file <none>
I20260806 02:52:33.102886 1632 fs_manager.cc:359] Metadata directory not provided
I20260806 02:52:33.102996 1632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:52:33.103269 1632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:52:33.106459 1632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984707115702-26446-0/minicluster-data/master-0/data/instance:
uuid: "44f33db7dcfd40b8817c10daaa6e8729"
format_stamp: "Formatted at 2026-08-06 02:52:33 on dist-test-slave-zj8q"
server_key: "46e4312e0f6cba9145e8ee4d614f7df0"
server_key_iv: "d71081a24cc7f7683dea3b0e3323bae0"
server_key_version: "encryptionkey@0"
I20260806 02:52:33.107307 1632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984707115702-26446-0/minicluster-data/master-0/wal/instance:
uuid: "44f33db7dcfd40b8817c10daaa6e8729"
format_stamp: "Formatted at 2026-08-06 02:52:33 on dist-test-slave-zj8q"
server_key: "46e4312e0f6cba9145e8ee4d614f7df0"
server_key_iv: "d71081a24cc7f7683dea3b0e3323bae0"
server_key_version: "encryptionkey@0"
I20260806 02:52:33.112942 1632 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 02:52:33.116798 1652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:33.118386 1632 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:52:33.118592 1632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984707115702-26446-0/minicluster-data/master-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984707115702-26446-0/minicluster-data/master-0/wal
uuid: "44f33db7dcfd40b8817c10daaa6e8729"
format_stamp: "Formatted at 2026-08-06 02:52:33 on dist-test-slave-zj8q"
server_key: "46e4312e0f6cba9145e8ee4d614f7df0"
server_key_iv: "d71081a24cc7f7683dea3b0e3323bae0"
server_key_version: "encryptionkey@0"
I20260806 02:52:33.118840 1632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984707115702-26446-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984707115702-26446-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984707115702-26446-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:52:33.145270 1632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:52:33.146334 1632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:52:33.146658 1632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:52:33.165140 1632 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.190:39709
I20260806 02:52:33.165161 1733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.190:39709 every 8 connection(s)
I20260806 02:52:33.166640 1632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
I20260806 02:52:33.170653 1734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:33.172462 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 1632
I20260806 02:52:33.172695 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984707115702-26446-0/minicluster-data/master-0/wal/instance
I20260806 02:52:33.173520 26446 external_mini_cluster.cc:1428] Setting key 6cce1b04254690bb6fc2c4674b6557da
I20260806 02:52:33.183995 1734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44f33db7dcfd40b8817c10daaa6e8729: Bootstrap starting.
I20260806 02:52:33.187932 1734 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44f33db7dcfd40b8817c10daaa6e8729: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:33.189225 1734 log.cc:824] T 00000000000000000000000000000000 P 44f33db7dcfd40b8817c10daaa6e8729: Log is configured to *not* fsync() on all Append() calls
I20260806 02:52:33.189677 1737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:33.175623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56886 (local address 127.25.211.190:39709)
0806 02:52:33.176802 (+ 1179us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:33.176815 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:33.176865 (+ 50us) server_negotiation.cc:407] Connection header received
0806 02:52:33.178725 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:33.178753 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:33.179273 (+ 520us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:33.179639 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:33.180370 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:33.182246 (+ 1876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:33.184310 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:33.185095 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:33.185280 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:33.186863 (+ 1583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:33.186873 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:33.186892 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:52:33.187506 (+ 614us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:33.187638 (+ 132us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:52:33.187668 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:33.187817 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:33.188155 (+ 338us) server_negotiation.cc:299] Negotiation successful
0806 02:52:33.188572 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":678,"thread_start_us":423,"threads_started":1}
I20260806 02:52:33.192691 1734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44f33db7dcfd40b8817c10daaa6e8729: No bootstrap required, opened a new log
I20260806 02:52:33.194056 26446 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:52:33.201216 1734 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44f33db7dcfd40b8817c10daaa6e8729 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44f33db7dcfd40b8817c10daaa6e8729" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 39709 } }
I20260806 02:52:33.201695 1734 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44f33db7dcfd40b8817c10daaa6e8729 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:33.201797 1734 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44f33db7dcfd40b8817c10daaa6e8729 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44f33db7dcfd40b8817c10daaa6e8729, State: Initialized, Role: FOLLOWER
I20260806 02:52:33.202519 1734 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44f33db7dcfd40b8817c10daaa6e8729 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44f33db7dcfd40b8817c10daaa6e8729" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 39709 } }
I20260806 02:52:33.202863 1734 raft_consensus.cc:388] T 00000000000000000000000000000000 P 44f33db7dcfd40b8817c10daaa6e8729 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:52:33.203020 1734 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44f33db7dcfd40b8817c10daaa6e8729 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:52:33.203246 1734 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44f33db7dcfd40b8817c10daaa6e8729 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:33.205732 1734 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44f33db7dcfd40b8817c10daaa6e8729 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44f33db7dcfd40b8817c10daaa6e8729" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 39709 } }
I20260806 02:52:33.206460 1734 leader_election.cc:302] T 00000000000000000000000000000000 P 44f33db7dcfd40b8817c10daaa6e8729 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44f33db7dcfd40b8817c10daaa6e8729; no voters:
I20260806 02:52:33.207152 1734 leader_election.cc:288] T 00000000000000000000000000000000 P 44f33db7dcfd40b8817c10daaa6e8729 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:52:33.207290 1739 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44f33db7dcfd40b8817c10daaa6e8729 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:52:33.208621 1739 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44f33db7dcfd40b8817c10daaa6e8729 [term 1 LEADER]: Becoming Leader. State: Replica: 44f33db7dcfd40b8817c10daaa6e8729, State: Running, Role: LEADER
I20260806 02:52:33.209203 1739 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44f33db7dcfd40b8817c10daaa6e8729 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44f33db7dcfd40b8817c10daaa6e8729" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 39709 } }
I20260806 02:52:33.209640 1734 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44f33db7dcfd40b8817c10daaa6e8729 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:52:33.212327 1741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44f33db7dcfd40b8817c10daaa6e8729 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44f33db7dcfd40b8817c10daaa6e8729. Latest consensus state: current_term: 1 leader_uuid: "44f33db7dcfd40b8817c10daaa6e8729" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44f33db7dcfd40b8817c10daaa6e8729" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 39709 } } }
I20260806 02:52:33.212785 1741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44f33db7dcfd40b8817c10daaa6e8729 [sys.catalog]: This master's current role is: LEADER
I20260806 02:52:33.213310 1740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44f33db7dcfd40b8817c10daaa6e8729 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44f33db7dcfd40b8817c10daaa6e8729" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44f33db7dcfd40b8817c10daaa6e8729" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 39709 } } }
I20260806 02:52:33.213658 1740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44f33db7dcfd40b8817c10daaa6e8729 [sys.catalog]: This master's current role is: LEADER
I20260806 02:52:33.214596 1750 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:52:33.220069 1750 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:52:33.230541 1750 catalog_manager.cc:1347] Generated new cluster ID: bd0f1cd60a824287bdb5908d9832bcda
I20260806 02:52:33.230679 1750 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:52:33.239828 1750 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:52:33.241405 1750 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:52:33.247831 1750 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 44f33db7dcfd40b8817c10daaa6e8729: Generated new TSK 0
I20260806 02:52:33.248819 1750 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260806 02:52:33.291863 1770 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260806 02:52:33.292090 1770 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260806 02:52:33.294330 1737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:33.269264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56900 (local address 127.25.211.190:39709)
0806 02:52:33.269495 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:33.269501 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:33.270138 (+ 637us) server_negotiation.cc:407] Connection header received
0806 02:52:33.270588 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:33.270595 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:33.270694 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:33.270875 (+ 181us) server_negotiation.cc:226] Negotiated authn=JWT
0806 02:52:33.271471 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:33.272610 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:33.274087 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:33.274802 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:33.274947 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:33.275570 (+ 623us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 02:52:33.293549 (+ 17979us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44833/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0806 02:52:33.293987 (+ 438us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56900: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44833/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":66}
W20260806 02:52:33.294458 1737 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56900: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44833/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260806 02:52:33.295733 1769 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:52:33.269220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:39709 (local address 127.0.0.1:56900)
0806 02:52:33.270024 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:33.270043 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:33.270277 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:33.270942 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:33.270946 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:33.270957 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0806 02:52:33.271311 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:33.271321 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:33.272778 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:33.272783 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:33.273924 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:33.273937 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:33.275013 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:33.275019 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:33.275167 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:33.275241 (+ 74us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 02:52:33.293896 (+ 18655us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44833/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0806 02:52:33.294230 (+ 334us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.25.211.190:39709: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44833/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":351,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.25.211.190:39709: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44833/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260806 02:52:33.297187 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 1632
I20260806 02:52:33.298457 1728 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:52:33.392902 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 1632
2026-08-06T02:52:33Z chronyd exiting
I20260806 02:52:33.452368 26446 test_util.cc:175] -----------------------------------------------
I20260806 02:52:33.452517 26446 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1785984707115702-26446-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (781 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260806 02:52:33.457419 26446 mini_oidc.cc:152] Starting JWKS server
I20260806 02:52:33.459446 26446 webserver.cc:460] Webserver started at https://127.0.0.1:43807/ using document root <none> and password file <none>
I20260806 02:52:33.459676 26446 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 02:52:33.460366 26446 webserver.cc:460] Webserver started at http://127.0.0.1:34639/ using document root <none> and password file <none>
2026-08-06T02:52:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:52:33Z Disabled control of system clock
I20260806 02:52:33.496779 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984707115702-26446-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984707115702-26446-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984707115702-26446-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:35619
--webserver_interface=127.25.211.190
--webserver_port=0
--builtin_ntp_servers=127.25.211.148:38633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.211.190:35619
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43807/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984707115702-26446-0/testchainca.cert with env {}
W20260806 02:52:33.845844 1782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:52:33.846204 1782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:52:33.846344 1782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:52:33.853889 1782 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:52:33.854040 1782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:52:33.854101 1782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:52:33.854209 1782 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:52:33.854274 1782 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 02:52:33.854362 1782 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 02:52:33.854417 1782 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43807/jwks.json
I20260806 02:52:33.865239 1782 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:38633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984707115702-26446-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984707115702-26446-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.211.190:35619
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:35619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43807/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.211.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984707115702-26446-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984707115702-26446-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:52:33.866762 1782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:52:33.868201 1782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:52:33.880704 1787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:33.880676 1791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:33.881227 1789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:52:33.881492 1782 server_base.cc:1034] running on GCE node
I20260806 02:52:33.882517 1782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:52:33.884021 1782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:52:33.885231 1782 hybrid_clock.cc:648] HybridClock initialized: now 1785984753885188 us; error 33 us; skew 500 ppm
I20260806 02:52:33.885609 1782 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:52:33.893273 1782 webserver.cc:460] Webserver started at http://127.25.211.190:46621/ using document root <none> and password file <none>
I20260806 02:52:33.894115 1782 fs_manager.cc:359] Metadata directory not provided
I20260806 02:52:33.894276 1782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:52:33.894623 1782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:52:33.897691 1782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984707115702-26446-0/minicluster-data/master-0/data/instance:
uuid: "51cc15c795ab426c9c11a4020db72adf"
format_stamp: "Formatted at 2026-08-06 02:52:33 on dist-test-slave-zj8q"
server_key: "805b8805360a25223b671faaa7d4f138"
server_key_iv: "e8a28795d3bd8664a069f8485bbc9b97"
server_key_version: "encryptionkey@0"
I20260806 02:52:33.898581 1782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984707115702-26446-0/minicluster-data/master-0/wal/instance:
uuid: "51cc15c795ab426c9c11a4020db72adf"
format_stamp: "Formatted at 2026-08-06 02:52:33 on dist-test-slave-zj8q"
server_key: "805b8805360a25223b671faaa7d4f138"
server_key_iv: "e8a28795d3bd8664a069f8485bbc9b97"
server_key_version: "encryptionkey@0"
I20260806 02:52:33.904083 1782 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 02:52:33.907747 1802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:33.909191 1782 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 02:52:33.909375 1782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984707115702-26446-0/minicluster-data/master-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984707115702-26446-0/minicluster-data/master-0/wal
uuid: "51cc15c795ab426c9c11a4020db72adf"
format_stamp: "Formatted at 2026-08-06 02:52:33 on dist-test-slave-zj8q"
server_key: "805b8805360a25223b671faaa7d4f138"
server_key_iv: "e8a28795d3bd8664a069f8485bbc9b97"
server_key_version: "encryptionkey@0"
I20260806 02:52:33.909591 1782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984707115702-26446-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984707115702-26446-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984707115702-26446-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:52:33.927122 1782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:52:33.928031 1782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:52:33.928339 1782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:52:33.945977 1782 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.190:35619
I20260806 02:52:33.945998 1881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.190:35619 every 8 connection(s)
I20260806 02:52:33.947466 1782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
I20260806 02:52:33.951421 1883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:33.952324 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 1782
I20260806 02:52:33.952499 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1785984707115702-26446-0/minicluster-data/master-0/wal/instance
I20260806 02:52:33.953151 26446 external_mini_cluster.cc:1428] Setting key aa71a22f1c200f08114d35808dfedb12
I20260806 02:52:33.966624 1883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51cc15c795ab426c9c11a4020db72adf: Bootstrap starting.
I20260806 02:52:33.968647 1887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:33.955091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42446 (local address 127.25.211.190:35619)
0806 02:52:33.956234 (+ 1143us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:33.956250 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:33.956306 (+ 56us) server_negotiation.cc:407] Connection header received
0806 02:52:33.958223 (+ 1917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:33.958257 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:33.958902 (+ 645us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:33.959393 (+ 491us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:33.959943 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:33.961819 (+ 1876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:33.963224 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:33.964344 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:33.964519 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:33.965659 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:33.965671 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:33.965691 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:52:33.966463 (+ 772us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:33.966577 (+ 114us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:52:33.966612 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:33.966734 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:33.966930 (+ 196us) server_negotiation.cc:299] Negotiation successful
0806 02:52:33.967304 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":322,"threads_started":1}
I20260806 02:52:33.970793 1883 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51cc15c795ab426c9c11a4020db72adf: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:33.972023 1883 log.cc:824] T 00000000000000000000000000000000 P 51cc15c795ab426c9c11a4020db72adf: Log is configured to *not* fsync() on all Append() calls
I20260806 02:52:33.972808 26446 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:52:33.974988 1883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51cc15c795ab426c9c11a4020db72adf: No bootstrap required, opened a new log
I20260806 02:52:33.982808 1883 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51cc15c795ab426c9c11a4020db72adf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51cc15c795ab426c9c11a4020db72adf" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 35619 } }
I20260806 02:52:33.983238 1883 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51cc15c795ab426c9c11a4020db72adf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:33.983351 1883 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51cc15c795ab426c9c11a4020db72adf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51cc15c795ab426c9c11a4020db72adf, State: Initialized, Role: FOLLOWER
I20260806 02:52:33.983948 1883 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51cc15c795ab426c9c11a4020db72adf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51cc15c795ab426c9c11a4020db72adf" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 35619 } }
I20260806 02:52:33.984184 1883 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51cc15c795ab426c9c11a4020db72adf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:52:33.984321 1883 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51cc15c795ab426c9c11a4020db72adf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:52:33.984477 1883 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51cc15c795ab426c9c11a4020db72adf [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:33.986997 1883 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51cc15c795ab426c9c11a4020db72adf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51cc15c795ab426c9c11a4020db72adf" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 35619 } }
I20260806 02:52:33.987561 1883 leader_election.cc:302] T 00000000000000000000000000000000 P 51cc15c795ab426c9c11a4020db72adf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51cc15c795ab426c9c11a4020db72adf; no voters:
I20260806 02:52:33.988198 1883 leader_election.cc:288] T 00000000000000000000000000000000 P 51cc15c795ab426c9c11a4020db72adf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:52:33.988346 1889 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51cc15c795ab426c9c11a4020db72adf [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:52:33.989321 1889 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51cc15c795ab426c9c11a4020db72adf [term 1 LEADER]: Becoming Leader. State: Replica: 51cc15c795ab426c9c11a4020db72adf, State: Running, Role: LEADER
I20260806 02:52:33.989830 1889 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51cc15c795ab426c9c11a4020db72adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51cc15c795ab426c9c11a4020db72adf" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 35619 } }
I20260806 02:52:33.990494 1883 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51cc15c795ab426c9c11a4020db72adf [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:52:33.992722 1891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51cc15c795ab426c9c11a4020db72adf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51cc15c795ab426c9c11a4020db72adf. Latest consensus state: current_term: 1 leader_uuid: "51cc15c795ab426c9c11a4020db72adf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51cc15c795ab426c9c11a4020db72adf" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 35619 } } }
I20260806 02:52:33.992710 1890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51cc15c795ab426c9c11a4020db72adf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "51cc15c795ab426c9c11a4020db72adf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51cc15c795ab426c9c11a4020db72adf" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 35619 } } }
I20260806 02:52:33.993146 1891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51cc15c795ab426c9c11a4020db72adf [sys.catalog]: This master's current role is: LEADER
I20260806 02:52:33.993225 1890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51cc15c795ab426c9c11a4020db72adf [sys.catalog]: This master's current role is: LEADER
I20260806 02:52:33.996534 1897 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:52:34.001835 1897 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:52:34.011317 1897 catalog_manager.cc:1347] Generated new cluster ID: f7f34a5d5f9943c5862f639d68f551f8
I20260806 02:52:34.011446 1897 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:52:34.033167 1897 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:52:34.034621 1897 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:52:34.043349 1897 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 51cc15c795ab426c9c11a4020db72adf: Generated new TSK 0
I20260806 02:52:34.044234 1897 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:52:34.100440 1887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:34.065113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42452 (local address 127.25.211.190:35619)
0806 02:52:34.065329 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:34.065336 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:34.065740 (+ 404us) server_negotiation.cc:407] Connection header received
0806 02:52:34.065997 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:34.066001 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:34.066081 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:34.066244 (+ 163us) server_negotiation.cc:226] Negotiated authn=JWT
0806 02:52:34.066854 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:34.068107 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:34.070288 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:34.070800 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:34.070966 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:34.071820 (+ 854us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 02:52:34.099650 (+ 27830us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43807/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0806 02:52:34.100087 (+ 437us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42452: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43807/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":83}
W20260806 02:52:34.100593 1887 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42452: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43807/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
E20260806 02:52:34.100960 1918 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260806 02:52:34.101068 1918 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260806 02:52:34.101166 1917 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:52:34.064900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:35619 (local address 127.0.0.1:42452)
0806 02:52:34.065610 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:34.065636 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:34.065804 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:34.066292 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:34.066298 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:34.066311 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0806 02:52:34.066634 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:34.066648 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:34.068313 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:34.068320 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:34.070077 (+ 1757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:34.070095 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:34.071206 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:34.071214 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:34.071428 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:34.071488 (+ 60us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 02:52:34.100154 (+ 28666us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43807/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0806 02:52:34.100416 (+ 262us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.25.211.190:35619: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43807/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":354,"threads_started":1}
I20260806 02:52:34.102216 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 1782
I20260806 02:52:34.103652 1874 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:52:34.208258 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 1782
2026-08-06T02:52:34Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (812 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260806 02:52:34.269595 26446 mini_oidc.cc:152] Starting JWKS server
I20260806 02:52:34.271613 26446 webserver.cc:460] Webserver started at https://127.0.0.1:36355/ using document root <none> and password file <none>
I20260806 02:52:34.271884 26446 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 02:52:34.272565 26446 webserver.cc:460] Webserver started at http://127.0.0.1:45165/ using document root <none> and password file <none>
2026-08-06T02:52:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:52:34Z Disabled control of system clock
I20260806 02:52:34.309134 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984707115702-26446-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984707115702-26446-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984707115702-26446-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:37119
--webserver_interface=127.25.211.190
--webserver_port=0
--builtin_ntp_servers=127.25.211.148:38189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.211.190:37119
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36355/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260806 02:52:34.650486 1926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:52:34.650832 1926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:52:34.650969 1926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:52:34.658407 1926 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:52:34.658565 1926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:52:34.658641 1926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:52:34.658723 1926 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:52:34.658782 1926 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 02:52:34.658867 1926 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 02:52:34.658921 1926 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36355/jwks.json
I20260806 02:52:34.672976 1926 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:38189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984707115702-26446-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984707115702-26446-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.211.190:37119
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:37119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36355/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.211.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984707115702-26446-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:52:34.676267 1926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:52:34.680827 1926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:52:34.708832 1933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:34.708596 1934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:34.709215 1936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:52:34.710662 1926 server_base.cc:1034] running on GCE node
I20260806 02:52:34.712872 1926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:52:34.715865 1926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:52:34.717284 1926 hybrid_clock.cc:648] HybridClock initialized: now 1785984754717155 us; error 95 us; skew 500 ppm
I20260806 02:52:34.718381 1926 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:52:34.723831 1926 webserver.cc:460] Webserver started at http://127.25.211.190:36937/ using document root <none> and password file <none>
I20260806 02:52:34.725597 1926 fs_manager.cc:359] Metadata directory not provided
I20260806 02:52:34.725872 1926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:52:34.726742 1926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:52:34.733145 1926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984707115702-26446-0/minicluster-data/master-0/data/instance:
uuid: "95c9840163e348b59c0a1ee36975f1d6"
format_stamp: "Formatted at 2026-08-06 02:52:34 on dist-test-slave-zj8q"
server_key: "8fe8749bdff13045ee7932ac2b99c4f8"
server_key_iv: "61eb9fd9178cac2c7333146c036e081e"
server_key_version: "encryptionkey@0"
I20260806 02:52:34.735333 1926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984707115702-26446-0/minicluster-data/master-0/wal/instance:
uuid: "95c9840163e348b59c0a1ee36975f1d6"
format_stamp: "Formatted at 2026-08-06 02:52:34 on dist-test-slave-zj8q"
server_key: "8fe8749bdff13045ee7932ac2b99c4f8"
server_key_iv: "61eb9fd9178cac2c7333146c036e081e"
server_key_version: "encryptionkey@0"
I20260806 02:52:34.751153 1926 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.005s sys 0.008s
I20260806 02:52:34.760016 1943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:34.763757 1926 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.006s sys 0.000s
I20260806 02:52:34.764219 1926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984707115702-26446-0/minicluster-data/master-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984707115702-26446-0/minicluster-data/master-0/wal
uuid: "95c9840163e348b59c0a1ee36975f1d6"
format_stamp: "Formatted at 2026-08-06 02:52:34 on dist-test-slave-zj8q"
server_key: "8fe8749bdff13045ee7932ac2b99c4f8"
server_key_iv: "61eb9fd9178cac2c7333146c036e081e"
server_key_version: "encryptionkey@0"
I20260806 02:52:34.764708 1926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984707115702-26446-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984707115702-26446-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984707115702-26446-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:52:34.802974 1926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:52:34.805013 1926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:52:34.805903 1926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:52:34.852411 1926 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.190:37119
I20260806 02:52:34.852447 2017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.190:37119 every 8 connection(s)
I20260806 02:52:34.855729 1926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
I20260806 02:52:34.858779 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 1926
I20260806 02:52:34.859546 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1785984707115702-26446-0/minicluster-data/master-0/wal/instance
I20260806 02:52:34.862135 26446 external_mini_cluster.cc:1428] Setting key a5c25eb1f5db1a6fc453188601b3eed2
I20260806 02:52:34.869879 2018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:34.895146 2021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:34.868037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52338 (local address 127.25.211.190:37119)
0806 02:52:34.870670 (+ 2633us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:34.870698 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:34.870870 (+ 172us) server_negotiation.cc:407] Connection header received
0806 02:52:34.875323 (+ 4453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:34.875383 (+ 60us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:34.876482 (+ 1099us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:34.877410 (+ 928us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:34.878812 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:34.882126 (+ 3314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:34.885297 (+ 3171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:34.887057 (+ 1760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:34.887477 (+ 420us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:34.890063 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:34.890083 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:34.890112 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:52:34.891316 (+ 1204us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:34.891486 (+ 170us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:52:34.891532 (+ 46us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:34.891751 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:34.892293 (+ 542us) server_negotiation.cc:299] Negotiation successful
0806 02:52:34.893005 (+ 712us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1414,"thread_start_us":925,"threads_started":1}
I20260806 02:52:34.895644 2018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95c9840163e348b59c0a1ee36975f1d6: Bootstrap starting.
I20260806 02:52:34.904165 26446 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:52:34.904215 2018 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95c9840163e348b59c0a1ee36975f1d6: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:34.906706 2018 log.cc:824] T 00000000000000000000000000000000 P 95c9840163e348b59c0a1ee36975f1d6: Log is configured to *not* fsync() on all Append() calls
I20260806 02:52:34.914420 2018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95c9840163e348b59c0a1ee36975f1d6: No bootstrap required, opened a new log
I20260806 02:52:34.933483 2018 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95c9840163e348b59c0a1ee36975f1d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95c9840163e348b59c0a1ee36975f1d6" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 37119 } }
I20260806 02:52:34.934377 2018 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95c9840163e348b59c0a1ee36975f1d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:34.934623 2018 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95c9840163e348b59c0a1ee36975f1d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95c9840163e348b59c0a1ee36975f1d6, State: Initialized, Role: FOLLOWER
I20260806 02:52:34.935878 2018 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95c9840163e348b59c0a1ee36975f1d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95c9840163e348b59c0a1ee36975f1d6" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 37119 } }
I20260806 02:52:34.936450 2018 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95c9840163e348b59c0a1ee36975f1d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:52:34.936658 2018 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95c9840163e348b59c0a1ee36975f1d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:52:34.936937 2018 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95c9840163e348b59c0a1ee36975f1d6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:34.941663 2018 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95c9840163e348b59c0a1ee36975f1d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95c9840163e348b59c0a1ee36975f1d6" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 37119 } }
I20260806 02:52:34.943073 2018 leader_election.cc:302] T 00000000000000000000000000000000 P 95c9840163e348b59c0a1ee36975f1d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95c9840163e348b59c0a1ee36975f1d6; no voters:
I20260806 02:52:34.944307 2018 leader_election.cc:288] T 00000000000000000000000000000000 P 95c9840163e348b59c0a1ee36975f1d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:52:34.944496 2023 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95c9840163e348b59c0a1ee36975f1d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:52:34.945874 2023 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95c9840163e348b59c0a1ee36975f1d6 [term 1 LEADER]: Becoming Leader. State: Replica: 95c9840163e348b59c0a1ee36975f1d6, State: Running, Role: LEADER
I20260806 02:52:34.947168 2023 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95c9840163e348b59c0a1ee36975f1d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95c9840163e348b59c0a1ee36975f1d6" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 37119 } }
I20260806 02:52:34.948586 2018 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95c9840163e348b59c0a1ee36975f1d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:52:34.957748 2025 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95c9840163e348b59c0a1ee36975f1d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95c9840163e348b59c0a1ee36975f1d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95c9840163e348b59c0a1ee36975f1d6" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 37119 } } }
I20260806 02:52:34.958251 2026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95c9840163e348b59c0a1ee36975f1d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95c9840163e348b59c0a1ee36975f1d6. Latest consensus state: current_term: 1 leader_uuid: "95c9840163e348b59c0a1ee36975f1d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95c9840163e348b59c0a1ee36975f1d6" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 37119 } } }
I20260806 02:52:34.959618 2026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95c9840163e348b59c0a1ee36975f1d6 [sys.catalog]: This master's current role is: LEADER
I20260806 02:52:34.959628 2025 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95c9840163e348b59c0a1ee36975f1d6 [sys.catalog]: This master's current role is: LEADER
I20260806 02:52:34.967675 2036 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:52:34.980573 2036 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:52:35.002061 2036 catalog_manager.cc:1347] Generated new cluster ID: 1dd59f0d88484cd0ba16cc3b19997737
I20260806 02:52:35.002452 2036 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:52:35.036554 2036 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:52:35.040117 2036 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:52:35.055274 2036 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95c9840163e348b59c0a1ee36975f1d6: Generated new TSK 0
I20260806 02:52:35.057641 2036 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:52:35.123566 2021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:35.100343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52346 (local address 127.25.211.190:37119)
0806 02:52:35.100889 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:35.100899 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:35.101428 (+ 529us) server_negotiation.cc:407] Connection header received
0806 02:52:35.101884 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:35.101904 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:35.102229 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:35.102708 (+ 479us) server_negotiation.cc:226] Negotiated authn=JWT
0806 02:52:35.104592 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:35.107431 (+ 2839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:35.111010 (+ 3579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:35.113136 (+ 2126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:35.113636 (+ 500us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:35.115039 (+ 1403us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 02:52:35.121591 (+ 6552us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36355/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0806 02:52:35.122694 (+ 1103us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52346: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36355/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":285}
W20260806 02:52:35.124475 2021 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52346: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36355/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260806 02:52:35.125713 2057 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:52:35.099866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:37119 (local address 127.0.0.1:52346)
0806 02:52:35.101196 (+ 1330us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:35.101225 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:35.101528 (+ 303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:35.102802 (+ 1274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:35.102831 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:35.102914 (+ 83us) client_negotiation.cc:189] Negotiated authn=JWT
0806 02:52:35.103997 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:35.104046 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:35.107903 (+ 3857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:35.107910 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:35.110599 (+ 2689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:35.110662 (+ 63us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:35.113783 (+ 3121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:35.113798 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:35.114215 (+ 417us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:35.114321 (+ 106us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 02:52:35.122320 (+ 7999us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36355/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0806 02:52:35.124063 (+ 1743us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.25.211.190:37119: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36355/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":1070,"mutex_wait_us":1027,"thread_start_us":801,"threads_started":1}
I20260806 02:52:35.128396 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 1926
I20260806 02:52:35.131693 2012 generic_service.cc:196] Checking for leaks (request via RPC)
E20260806 02:52:35.150194 2059 webserver.cc:533] Webserver: sslize failed:
E20260806 02:52:35.150578 2059 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260806 02:52:35.304075 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 1926
2026-08-06T02:52:35Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (1112 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-06T02:52:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:52:35Z Disabled control of system clock
I20260806 02:52:35.443224 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984707115702-26446-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984707115702-26446-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984707115702-26446-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:46579
--webserver_interface=127.25.211.190
--webserver_port=0
--builtin_ntp_servers=127.25.211.148:39669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.211.190:46579
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984707115702-26446-0/testchainca.cert with env {}
W20260806 02:52:35.861045 2068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:52:35.861389 2068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:52:35.861521 2068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:52:35.869257 2068 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:52:35.869426 2068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:52:35.869506 2068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:52:35.869593 2068 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:52:35.869658 2068 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 02:52:35.869750 2068 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 02:52:35.869805 2068 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260806 02:52:35.880332 2068 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:39669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984707115702-26446-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984707115702-26446-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.211.190:46579
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:46579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.211.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984707115702-26446-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984707115702-26446-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:52:35.881750 2068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:52:35.883163 2068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:52:35.900403 2078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:35.900316 2082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:35.900375 2080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:52:35.901546 2068 server_base.cc:1034] running on GCE node
I20260806 02:52:35.902535 2068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:52:35.904083 2068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:52:35.905294 2068 hybrid_clock.cc:648] HybridClock initialized: now 1785984755905242 us; error 48 us; skew 500 ppm
I20260806 02:52:35.905678 2068 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:52:35.908095 2068 webserver.cc:460] Webserver started at http://127.25.211.190:41229/ using document root <none> and password file <none>
I20260806 02:52:35.908835 2068 fs_manager.cc:359] Metadata directory not provided
I20260806 02:52:35.908970 2068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:52:35.909281 2068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:52:35.912339 2068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984707115702-26446-0/minicluster-data/master-0/data/instance:
uuid: "c4c31cf25f704ac888dc2d48c75db35c"
format_stamp: "Formatted at 2026-08-06 02:52:35 on dist-test-slave-zj8q"
server_key: "2f121cbf2c1eab9218e2ce11bd8b2db4"
server_key_iv: "1a454e97ecd933a75e32b0ce89b96b68"
server_key_version: "encryptionkey@0"
I20260806 02:52:35.913139 2068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984707115702-26446-0/minicluster-data/master-0/wal/instance:
uuid: "c4c31cf25f704ac888dc2d48c75db35c"
format_stamp: "Formatted at 2026-08-06 02:52:35 on dist-test-slave-zj8q"
server_key: "2f121cbf2c1eab9218e2ce11bd8b2db4"
server_key_iv: "1a454e97ecd933a75e32b0ce89b96b68"
server_key_version: "encryptionkey@0"
I20260806 02:52:35.918674 2068 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260806 02:52:35.922459 2091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:35.923918 2068 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:52:35.924108 2068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984707115702-26446-0/minicluster-data/master-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984707115702-26446-0/minicluster-data/master-0/wal
uuid: "c4c31cf25f704ac888dc2d48c75db35c"
format_stamp: "Formatted at 2026-08-06 02:52:35 on dist-test-slave-zj8q"
server_key: "2f121cbf2c1eab9218e2ce11bd8b2db4"
server_key_iv: "1a454e97ecd933a75e32b0ce89b96b68"
server_key_version: "encryptionkey@0"
I20260806 02:52:35.924329 2068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984707115702-26446-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984707115702-26446-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984707115702-26446-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:52:35.936137 2068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:52:35.937002 2068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:52:35.937290 2068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:52:35.955358 2068 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.190:46579
I20260806 02:52:35.955379 2165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.190:46579 every 8 connection(s)
I20260806 02:52:35.956795 2068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
I20260806 02:52:35.960317 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 2068
I20260806 02:52:35.960515 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1785984707115702-26446-0/minicluster-data/master-0/wal/instance
I20260806 02:52:35.960942 2166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:35.961290 26446 external_mini_cluster.cc:1428] Setting key 05383695063481b832c8e43b97a1079e
I20260806 02:52:35.975216 2166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4c31cf25f704ac888dc2d48c75db35c: Bootstrap starting.
I20260806 02:52:35.975772 2170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:35.963212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44896 (local address 127.25.211.190:46579)
0806 02:52:35.964266 (+ 1054us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:35.964280 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:35.964327 (+ 47us) server_negotiation.cc:407] Connection header received
0806 02:52:35.966015 (+ 1688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:35.966048 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:35.966635 (+ 587us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:35.967018 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:35.967697 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:35.969086 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:35.970944 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:35.971678 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:35.971835 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:35.973117 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:35.973126 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:35.973143 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:52:35.973759 (+ 616us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:35.973846 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:52:35.973869 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:35.973967 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:35.974304 (+ 337us) server_negotiation.cc:299] Negotiation successful
0806 02:52:35.974681 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":364,"threads_started":1}
I20260806 02:52:35.978842 2166 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4c31cf25f704ac888dc2d48c75db35c: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:35.979998 26446 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:52:35.980141 2166 log.cc:824] T 00000000000000000000000000000000 P c4c31cf25f704ac888dc2d48c75db35c: Log is configured to *not* fsync() on all Append() calls
I20260806 02:52:35.983649 2166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4c31cf25f704ac888dc2d48c75db35c: No bootstrap required, opened a new log
I20260806 02:52:35.991633 2166 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4c31cf25f704ac888dc2d48c75db35c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4c31cf25f704ac888dc2d48c75db35c" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 46579 } }
I20260806 02:52:35.992045 2166 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4c31cf25f704ac888dc2d48c75db35c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:35.992125 2166 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4c31cf25f704ac888dc2d48c75db35c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4c31cf25f704ac888dc2d48c75db35c, State: Initialized, Role: FOLLOWER
I20260806 02:52:35.992687 2166 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4c31cf25f704ac888dc2d48c75db35c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4c31cf25f704ac888dc2d48c75db35c" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 46579 } }
I20260806 02:52:35.992906 2166 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4c31cf25f704ac888dc2d48c75db35c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:52:35.993006 2166 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4c31cf25f704ac888dc2d48c75db35c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:52:35.993144 2166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4c31cf25f704ac888dc2d48c75db35c [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:35.995484 2166 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4c31cf25f704ac888dc2d48c75db35c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4c31cf25f704ac888dc2d48c75db35c" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 46579 } }
I20260806 02:52:35.996014 2166 leader_election.cc:302] T 00000000000000000000000000000000 P c4c31cf25f704ac888dc2d48c75db35c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4c31cf25f704ac888dc2d48c75db35c; no voters:
I20260806 02:52:35.996639 2166 leader_election.cc:288] T 00000000000000000000000000000000 P c4c31cf25f704ac888dc2d48c75db35c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:52:35.996827 2172 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4c31cf25f704ac888dc2d48c75db35c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:52:35.997524 2172 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4c31cf25f704ac888dc2d48c75db35c [term 1 LEADER]: Becoming Leader. State: Replica: c4c31cf25f704ac888dc2d48c75db35c, State: Running, Role: LEADER
I20260806 02:52:35.998106 2172 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4c31cf25f704ac888dc2d48c75db35c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4c31cf25f704ac888dc2d48c75db35c" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 46579 } }
I20260806 02:52:35.998898 2166 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4c31cf25f704ac888dc2d48c75db35c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:52:36.001150 2181 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4c31cf25f704ac888dc2d48c75db35c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4c31cf25f704ac888dc2d48c75db35c. Latest consensus state: current_term: 1 leader_uuid: "c4c31cf25f704ac888dc2d48c75db35c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4c31cf25f704ac888dc2d48c75db35c" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 46579 } } }
I20260806 02:52:36.001111 2174 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4c31cf25f704ac888dc2d48c75db35c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4c31cf25f704ac888dc2d48c75db35c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4c31cf25f704ac888dc2d48c75db35c" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 46579 } } }
I20260806 02:52:36.001538 2174 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4c31cf25f704ac888dc2d48c75db35c [sys.catalog]: This master's current role is: LEADER
I20260806 02:52:36.001538 2181 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4c31cf25f704ac888dc2d48c75db35c [sys.catalog]: This master's current role is: LEADER
I20260806 02:52:36.002589 2186 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:52:36.008561 2186 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:52:36.019899 2186 catalog_manager.cc:1347] Generated new cluster ID: df1eb85252cd491e80271b3976bbf9a6
I20260806 02:52:36.020097 2186 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:52:36.046010 2186 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:52:36.047987 2186 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:52:36.058818 2186 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4c31cf25f704ac888dc2d48c75db35c: Generated new TSK 0
I20260806 02:52:36.060014 2186 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:52:36.157543 2170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:36.077077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44902 (local address 127.25.211.190:46579)
0806 02:52:36.077249 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:36.077254 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:36.077623 (+ 369us) server_negotiation.cc:407] Connection header received
0806 02:52:36.077885 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:36.077895 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:36.077998 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:36.078256 (+ 258us) server_negotiation.cc:226] Negotiated authn=JWT
0806 02:52:36.078702 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:36.079873 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:36.081330 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:36.082035 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:36.082201 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:36.082737 (+ 536us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 02:52:36.156760 (+ 74023us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0806 02:52:36.157222 (+ 462us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44902: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":50}
W20260806 02:52:36.157702 2170 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44902: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260806 02:52:36.157718 2216 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:52:36.076867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:46579 (local address 127.0.0.1:44902)
0806 02:52:36.077498 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:36.077516 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:36.077682 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:36.078268 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:36.078273 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:36.078285 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0806 02:52:36.078540 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:36.078549 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:36.080022 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:36.080026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:36.081172 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:36.081185 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:36.082242 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:36.082247 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:36.082391 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:36.082431 (+ 40us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 02:52:36.157120 (+ 74689us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0806 02:52:36.157387 (+ 267us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.25.211.190:46579: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":316,"threads_started":1}
I20260806 02:52:36.159245 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 2068
I20260806 02:52:36.160540 2160 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:52:36.261961 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 2068
2026-08-06T02:52:36Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (920 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (422 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (432 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:52:37 dist-test-slave-zj8q krb5kdc[2239](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:52:37 dist-test-slave-zj8q krb5kdc[2239](info): set up 2 sockets
Aug 06 02:52:37 dist-test-slave-zj8q krb5kdc[2239](info): commencing operation
krb5kdc: starting...
W20260806 02:52:38.330868 26446 mini_kdc.cc:121] Time spent starting KDC: real 1.172s user 0.003s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Aug 06 02:52:38 dist-test-slave-zj8q krb5kdc[2239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:52:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:52:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.211.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab.
Entry for principal kudu/127.25.211.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab.
I20260806 02:52:38.593879 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:39327
--webserver_interface=127.25.211.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:40321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.211.190:39327
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:52:38.935621 2265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:52:38.935962 2265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:52:38.936100 2265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:52:38.943462 2265 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:52:38.943607 2265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:52:38.943686 2265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:52:38.943773 2265 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:52:38.943835 2265 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:52:38.954231 2265 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:40321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.211.190:39327
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:39327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.211.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:52:38.955672 2265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:52:38.957041 2265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:52:38.969499 2274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:38.969532 2275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:38.969466 2278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:52:38.971112 2265 server_base.cc:1034] running on GCE node
I20260806 02:52:38.971923 2265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:52:38.973376 2265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:52:38.974587 2265 hybrid_clock.cc:648] HybridClock initialized: now 1785984758974543 us; error 29 us; skew 500 ppm
I20260806 02:52:38.974942 2265 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:52:38 dist-test-slave-zj8q krb5kdc[2239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984758, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:52:38.984191 2265 init.cc:377] Logged in from keytab as kudu/127.25.211.190@KRBTEST.COM (short username kudu)
I20260806 02:52:38.985770 2265 webserver.cc:460] Webserver started at http://127.25.211.190:38953/ using document root <none> and password file <none>
I20260806 02:52:38.986546 2265 fs_manager.cc:359] Metadata directory not provided
I20260806 02:52:38.986666 2265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:52:38.986950 2265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:52:38.990020 2265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/master-0/data/instance:
uuid: "31064033f0e04b788c52625485e71c46"
format_stamp: "Formatted at 2026-08-06 02:52:38 on dist-test-slave-zj8q"
server_key: "13430793737a1dafd4444d644372d0dd"
server_key_iv: "4a6ce562d5277884566bdebc2523b5e2"
server_key_version: "encryptionkey@0"
I20260806 02:52:38.990863 2265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/master-0/wal/instance:
uuid: "31064033f0e04b788c52625485e71c46"
format_stamp: "Formatted at 2026-08-06 02:52:38 on dist-test-slave-zj8q"
server_key: "13430793737a1dafd4444d644372d0dd"
server_key_iv: "4a6ce562d5277884566bdebc2523b5e2"
server_key_version: "encryptionkey@0"
I20260806 02:52:38.996248 2265 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 02:52:39.000337 2289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:39.002023 2265 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:52:39.002272 2265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/master-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/master-0/wal
uuid: "31064033f0e04b788c52625485e71c46"
format_stamp: "Formatted at 2026-08-06 02:52:38 on dist-test-slave-zj8q"
server_key: "13430793737a1dafd4444d644372d0dd"
server_key_iv: "4a6ce562d5277884566bdebc2523b5e2"
server_key_version: "encryptionkey@0"
I20260806 02:52:39.002493 2265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:52:39.025908 2265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:52:39.030666 2265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:52:39.030968 2265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:52:39.049875 2265 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.190:39327
I20260806 02:52:39.049907 2373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.190:39327 every 8 connection(s)
I20260806 02:52:39.051337 2265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
I20260806 02:52:39.055423 2374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:39.060066 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 2265
I20260806 02:52:39.060268 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/master-0/wal/instance
I20260806 02:52:39.060932 26446 external_mini_cluster.cc:1428] Setting key 39692db959503785fe6e674e6958faf7
I20260806 02:52:39.068429 2374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31064033f0e04b788c52625485e71c46: Bootstrap starting.
I20260806 02:52:39.072502 2374 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31064033f0e04b788c52625485e71c46: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:39.073957 2374 log.cc:824] T 00000000000000000000000000000000 P 31064033f0e04b788c52625485e71c46: Log is configured to *not* fsync() on all Append() calls
Aug 06 02:52:39 dist-test-slave-zj8q krb5kdc[2239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:52:39.077298 2374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31064033f0e04b788c52625485e71c46: No bootstrap required, opened a new log
I20260806 02:52:39.086480 2379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:39.063032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60740 (local address 127.25.211.190:39327)
0806 02:52:39.063878 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:39.063898 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:39.063955 (+ 57us) server_negotiation.cc:407] Connection header received
0806 02:52:39.066579 (+ 2624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:39.066619 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:39.067230 (+ 611us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:39.067708 (+ 478us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:39.069234 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:39.071367 (+ 2133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:39.072868 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:39.073995 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:39.074254 (+ 259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:39.078715 (+ 4461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:39.078725 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:39.078743 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:39.078799 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:39.081974 (+ 3175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:39.082870 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:39.082879 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:39.082886 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:39.083071 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:39.083480 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:39.083484 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:39.083486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:39.084077 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:39.084326 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:39.084826 (+ 500us) server_negotiation.cc:299] Negotiation successful
0806 02:52:39.085283 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":363,"threads_started":1}
I20260806 02:52:39.086709 2374 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31064033f0e04b788c52625485e71c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31064033f0e04b788c52625485e71c46" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 39327 } }
I20260806 02:52:39.087169 2374 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31064033f0e04b788c52625485e71c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:39.087297 2374 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31064033f0e04b788c52625485e71c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31064033f0e04b788c52625485e71c46, State: Initialized, Role: FOLLOWER
I20260806 02:52:39.087925 2374 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31064033f0e04b788c52625485e71c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31064033f0e04b788c52625485e71c46" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 39327 } }
I20260806 02:52:39.088198 2374 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31064033f0e04b788c52625485e71c46 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:52:39.088333 2374 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31064033f0e04b788c52625485e71c46 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:52:39.088512 2374 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31064033f0e04b788c52625485e71c46 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:39.091081 2374 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31064033f0e04b788c52625485e71c46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31064033f0e04b788c52625485e71c46" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 39327 } }
I20260806 02:52:39.091647 2374 leader_election.cc:302] T 00000000000000000000000000000000 P 31064033f0e04b788c52625485e71c46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31064033f0e04b788c52625485e71c46; no voters:
I20260806 02:52:39.092276 2374 leader_election.cc:288] T 00000000000000000000000000000000 P 31064033f0e04b788c52625485e71c46 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:52:39.092404 2381 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31064033f0e04b788c52625485e71c46 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:52:39.093082 2381 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31064033f0e04b788c52625485e71c46 [term 1 LEADER]: Becoming Leader. State: Replica: 31064033f0e04b788c52625485e71c46, State: Running, Role: LEADER
I20260806 02:52:39.093753 2381 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31064033f0e04b788c52625485e71c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31064033f0e04b788c52625485e71c46" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 39327 } }
I20260806 02:52:39.094667 2374 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31064033f0e04b788c52625485e71c46 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:52:39.096982 2385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31064033f0e04b788c52625485e71c46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31064033f0e04b788c52625485e71c46. Latest consensus state: current_term: 1 leader_uuid: "31064033f0e04b788c52625485e71c46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31064033f0e04b788c52625485e71c46" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 39327 } } }
I20260806 02:52:39.097440 2385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31064033f0e04b788c52625485e71c46 [sys.catalog]: This master's current role is: LEADER
I20260806 02:52:39.097592 2382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31064033f0e04b788c52625485e71c46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31064033f0e04b788c52625485e71c46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31064033f0e04b788c52625485e71c46" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 39327 } } }
I20260806 02:52:39.099097 2382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31064033f0e04b788c52625485e71c46 [sys.catalog]: This master's current role is: LEADER
I20260806 02:52:39.099748 2392 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:52:39.107245 2392 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:52:39.116101 2392 catalog_manager.cc:1347] Generated new cluster ID: 32ffd816b56347ba84a2daacc9d40599
I20260806 02:52:39.116242 2392 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:52:39.132378 2392 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:52:39.134379 2392 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:52:39.144330 2392 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31064033f0e04b788c52625485e71c46: Generated new TSK 0
I20260806 02:52:39.145485 2392 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.211.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab.
Entry for principal kudu/127.25.211.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab.
I20260806 02:52:39.210644 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.129:0
--local_ip_for_outbound_sockets=127.25.211.129
--webserver_interface=127.25.211.129
--webserver_port=0
--tserver_master_addrs=127.25.211.190:39327
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab
--principal=kudu/127.25.211.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:40321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:52:39.558670 2414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:52:39.558979 2414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:52:39.559113 2414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:52:39.566552 2414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:52:39.566672 2414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:52:39.566840 2414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.129
I20260806 02:52:39.576916 2414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:40321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab
--principal=kudu/127.25.211.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.211.129
--webserver_port=0
--tserver_master_addrs=127.25.211.190:39327
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.129
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:52:39.578356 2414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:52:39.579779 2414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:52:39.593622 2423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:39.593720 2420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:39.593669 2421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:52:39.594134 2414 server_base.cc:1034] running on GCE node
I20260806 02:52:39.595048 2414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:52:39.596097 2414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:52:39.597316 2414 hybrid_clock.cc:648] HybridClock initialized: now 1785984759597262 us; error 40 us; skew 500 ppm
I20260806 02:52:39.597643 2414 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:52:39 dist-test-slave-zj8q krb5kdc[2239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984759, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:52:39.600904 2414 init.cc:377] Logged in from keytab as kudu/127.25.211.129@KRBTEST.COM (short username kudu)
I20260806 02:52:39.602497 2414 webserver.cc:460] Webserver started at http://127.25.211.129:41971/ using document root <none> and password file <none>
I20260806 02:52:39.603251 2414 fs_manager.cc:359] Metadata directory not provided
I20260806 02:52:39.603392 2414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:52:39.603724 2414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:52:39.606873 2414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/ts-0/data/instance:
uuid: "37b5fc503c1141bb8a7af29bce56ed6e"
format_stamp: "Formatted at 2026-08-06 02:52:39 on dist-test-slave-zj8q"
server_key: "b64a1d7eeff8bbb1dffc5040504e1083"
server_key_iv: "10a8b702e9c582b87eac9d23e27b3b37"
server_key_version: "encryptionkey@0"
I20260806 02:52:39.607705 2414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/ts-0/wal/instance:
uuid: "37b5fc503c1141bb8a7af29bce56ed6e"
format_stamp: "Formatted at 2026-08-06 02:52:39 on dist-test-slave-zj8q"
server_key: "b64a1d7eeff8bbb1dffc5040504e1083"
server_key_iv: "10a8b702e9c582b87eac9d23e27b3b37"
server_key_version: "encryptionkey@0"
I20260806 02:52:39.613098 2414 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 02:52:39.616725 2434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:39.618263 2414 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:52:39.618443 2414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/ts-0/wal
uuid: "37b5fc503c1141bb8a7af29bce56ed6e"
format_stamp: "Formatted at 2026-08-06 02:52:39 on dist-test-slave-zj8q"
server_key: "b64a1d7eeff8bbb1dffc5040504e1083"
server_key_iv: "10a8b702e9c582b87eac9d23e27b3b37"
server_key_version: "encryptionkey@0"
I20260806 02:52:39.618669 2414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:52:39.646888 2414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:52:39.651252 2414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:52:39.651539 2414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:52:39.652523 2414 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:52:39.653915 2414 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:52:39.653993 2414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:39.654093 2414 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:52:39.654137 2414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:39.684533 2414 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.129:40235
I20260806 02:52:39.684576 2606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.129:40235 every 8 connection(s)
I20260806 02:52:39.685920 2414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
I20260806 02:52:39.687928 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 2414
I20260806 02:52:39.688118 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/ts-0/wal/instance
I20260806 02:52:39.689023 26446 external_mini_cluster.cc:1428] Setting key 9c603754c5d2919bf5d67a6a7a643aa9
Aug 06 02:52:39 dist-test-slave-zj8q krb5kdc[2239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984759, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.129@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:52:39.708703 2612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:39.688888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:44579 (local address 127.25.211.190:39327)
0806 02:52:39.689367 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:39.689372 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:39.690659 (+ 1287us) server_negotiation.cc:407] Connection header received
0806 02:52:39.692682 (+ 2023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:39.692687 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:39.692760 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:39.692855 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:39.694763 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:39.695554 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:39.697219 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:39.697641 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:39.697773 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:39.701632 (+ 3859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:39.701639 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:39.701645 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:39.701693 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:39.704852 (+ 3159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:39.705832 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:39.705838 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:39.705841 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:39.706015 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:39.706645 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:39.706650 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:39.706653 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:39.707017 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:39.707186 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:39.708065 (+ 879us) server_negotiation.cc:299] Negotiation successful
0806 02:52:39.708345 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":249,"threads_started":1}
I20260806 02:52:39.709548 2613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:39.689580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:39327 (local address 127.25.211.129:44579)
0806 02:52:39.690464 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:39.690524 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:39.692399 (+ 1875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:39.693124 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:39.693133 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:39.693687 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:39.694475 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:39.694499 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:39.695722 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:39.695727 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:39.697053 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:39.697074 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:39.697886 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:39.697894 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:39.698188 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:39.698894 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:52:39.698919 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:39.701329 (+ 2410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:39.705054 (+ 3725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:39.705064 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:39.705081 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:39.705664 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:39.706191 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:39.706197 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:39.706201 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:39.706477 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:39.707182 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:39.707191 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:39.707590 (+ 399us) client_negotiation.cc:769] Sending connection context
0806 02:52:39.707868 (+ 278us) client_negotiation.cc:240] Negotiation successful
0806 02:52:39.708330 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":326,"threads_started":1}
I20260806 02:52:39.711295 2609 heartbeater.cc:347] Connected to a master server at 127.25.211.190:39327
I20260806 02:52:39.711684 2609 heartbeater.cc:464] Registering TS with master...
I20260806 02:52:39.712685 2609 heartbeater.cc:511] Master 127.25.211.190:39327 requested a full tablet report, sending...
I20260806 02:52:39.715361 2314 ts_manager.cc:194] Registered new tserver with Master: 37b5fc503c1141bb8a7af29bce56ed6e (127.25.211.129:40235)
WARNING: no policy specified for kudu/127.25.211.130@KRBTEST.COM; defaulting to no policy
I20260806 02:52:39.716990 2314 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.211.129@KRBTEST.COM'} at 127.25.211.129:44579
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab.
Entry for principal kudu/127.25.211.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab.
I20260806 02:52:39.749944 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.130:0
--local_ip_for_outbound_sockets=127.25.211.130
--webserver_interface=127.25.211.130
--webserver_port=0
--tserver_master_addrs=127.25.211.190:39327
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab
--principal=kudu/127.25.211.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:40321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:52:40.095778 2617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:52:40.096125 2617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:52:40.096272 2617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:52:40.103976 2617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:52:40.104130 2617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:52:40.104287 2617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.130
I20260806 02:52:40.114670 2617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:40321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab
--principal=kudu/127.25.211.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.211.130
--webserver_port=0
--tserver_master_addrs=127.25.211.190:39327
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.130
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:52:40.116072 2617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:52:40.117406 2617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:52:40.130707 2626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:40.130638 2628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:40.130658 2625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:52:40.131476 2617 server_base.cc:1034] running on GCE node
I20260806 02:52:40.132123 2617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:52:40.133211 2617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:52:40.134423 2617 hybrid_clock.cc:648] HybridClock initialized: now 1785984760134371 us; error 37 us; skew 500 ppm
I20260806 02:52:40.134783 2617 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:52:40 dist-test-slave-zj8q krb5kdc[2239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984760, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:52:40.138027 2617 init.cc:377] Logged in from keytab as kudu/127.25.211.130@KRBTEST.COM (short username kudu)
I20260806 02:52:40.139686 2617 webserver.cc:460] Webserver started at http://127.25.211.130:37379/ using document root <none> and password file <none>
I20260806 02:52:40.140429 2617 fs_manager.cc:359] Metadata directory not provided
I20260806 02:52:40.140551 2617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:52:40.140878 2617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:52:40.144033 2617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/ts-1/data/instance:
uuid: "875c9cee98db48d192fabc95b4cd904a"
format_stamp: "Formatted at 2026-08-06 02:52:40 on dist-test-slave-zj8q"
server_key: "147379115fc7703eb4d97128e6538c37"
server_key_iv: "d6c6d3acb402f6b7021ce44f58be75f1"
server_key_version: "encryptionkey@0"
I20260806 02:52:40.145000 2617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/ts-1/wal/instance:
uuid: "875c9cee98db48d192fabc95b4cd904a"
format_stamp: "Formatted at 2026-08-06 02:52:40 on dist-test-slave-zj8q"
server_key: "147379115fc7703eb4d97128e6538c37"
server_key_iv: "d6c6d3acb402f6b7021ce44f58be75f1"
server_key_version: "encryptionkey@0"
I20260806 02:52:40.150697 2617 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260806 02:52:40.154428 2645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:40.155939 2617 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:52:40.156131 2617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/ts-1/wal
uuid: "875c9cee98db48d192fabc95b4cd904a"
format_stamp: "Formatted at 2026-08-06 02:52:40 on dist-test-slave-zj8q"
server_key: "147379115fc7703eb4d97128e6538c37"
server_key_iv: "d6c6d3acb402f6b7021ce44f58be75f1"
server_key_version: "encryptionkey@0"
I20260806 02:52:40.156349 2617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:52:40.194710 2617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:52:40.198954 2617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:52:40.199247 2617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:52:40.200299 2617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:52:40.201757 2617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:52:40.201838 2617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:40.201972 2617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:52:40.202023 2617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:40.234760 2617 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.130:40051
I20260806 02:52:40.234772 2805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.130:40051 every 8 connection(s)
I20260806 02:52:40.236320 2617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
I20260806 02:52:40.236931 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 2617
I20260806 02:52:40.237114 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/ts-1/wal/instance
I20260806 02:52:40.237735 26446 external_mini_cluster.cc:1428] Setting key 3e59533b75ed5a149ef35b02cc79a61d
Aug 06 02:52:40 dist-test-slave-zj8q krb5kdc[2239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984760, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.130@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:52:40.257469 2808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:40.239002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:59763 (local address 127.25.211.190:39327)
0806 02:52:40.239435 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:40.239439 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:40.240702 (+ 1263us) server_negotiation.cc:407] Connection header received
0806 02:52:40.242637 (+ 1935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:40.242643 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:40.242722 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:40.242832 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:40.244307 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:40.245301 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:40.247022 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:40.247590 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:40.247710 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:40.251610 (+ 3900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:40.251615 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:40.251618 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:40.251653 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:40.254271 (+ 2618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:40.255052 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:40.255057 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:40.255060 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:40.255196 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:40.255647 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:40.255650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:40.255653 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:40.255935 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:40.256071 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:40.256840 (+ 769us) server_negotiation.cc:299] Negotiation successful
0806 02:52:40.257089 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":229,"threads_started":1}
I20260806 02:52:40.258646 2809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:40.239718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:39327 (local address 127.25.211.130:59763)
0806 02:52:40.240527 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:40.240573 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:40.242375 (+ 1802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:40.243027 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:40.243035 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:40.243569 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:40.244145 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:40.244164 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:40.245487 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:40.245492 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:40.246873 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:40.246888 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:40.247813 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:40.247819 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:40.248047 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:40.248697 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:52:40.248725 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:40.251365 (+ 2640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:40.254429 (+ 3064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:40.254439 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:40.254459 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:40.254920 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:40.255309 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:40.255315 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:40.255319 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:40.255528 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:40.256064 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:40.256072 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:40.256457 (+ 385us) client_negotiation.cc:769] Sending connection context
0806 02:52:40.256708 (+ 251us) client_negotiation.cc:240] Negotiation successful
0806 02:52:40.257336 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":334,"threads_started":1}
I20260806 02:52:40.260218 2806 heartbeater.cc:347] Connected to a master server at 127.25.211.190:39327
I20260806 02:52:40.260589 2806 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.25.211.131@KRBTEST.COM; defaulting to no policy
I20260806 02:52:40.261437 2806 heartbeater.cc:511] Master 127.25.211.190:39327 requested a full tablet report, sending...
I20260806 02:52:40.262959 2314 ts_manager.cc:194] Registered new tserver with Master: 875c9cee98db48d192fabc95b4cd904a (127.25.211.130:40051)
I20260806 02:52:40.263870 2314 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.211.130@KRBTEST.COM'} at 127.25.211.130:59763
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab.
Entry for principal kudu/127.25.211.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab.
I20260806 02:52:40.293852 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.131:0
--local_ip_for_outbound_sockets=127.25.211.131
--webserver_interface=127.25.211.131
--webserver_port=0
--tserver_master_addrs=127.25.211.190:39327
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab
--principal=kudu/127.25.211.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:40321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:52:40.640712 2815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:52:40.641008 2815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:52:40.641124 2815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:52:40.648638 2815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:52:40.648761 2815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:52:40.648907 2815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.131
I20260806 02:52:40.658954 2815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:40321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab
--principal=kudu/127.25.211.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.211.131
--webserver_port=0
--tserver_master_addrs=127.25.211.190:39327
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.131
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:52:40.660254 2815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:52:40.661702 2815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:52:40.674976 2821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:40.674907 2822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:40.674944 2824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:52:40.675911 2815 server_base.cc:1034] running on GCE node
I20260806 02:52:40.676561 2815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:52:40.677549 2815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:52:40.678758 2815 hybrid_clock.cc:648] HybridClock initialized: now 1785984760678700 us; error 52 us; skew 500 ppm
I20260806 02:52:40.679076 2815 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:52:40 dist-test-slave-zj8q krb5kdc[2239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984760, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:52:40.682442 2815 init.cc:377] Logged in from keytab as kudu/127.25.211.131@KRBTEST.COM (short username kudu)
I20260806 02:52:40.684022 2815 webserver.cc:460] Webserver started at http://127.25.211.131:41531/ using document root <none> and password file <none>
I20260806 02:52:40.684935 2815 fs_manager.cc:359] Metadata directory not provided
I20260806 02:52:40.685089 2815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:52:40.685499 2815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:52:40.689750 2815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/ts-2/data/instance:
uuid: "a2b1f777a6334e5996f0fe527734080a"
format_stamp: "Formatted at 2026-08-06 02:52:40 on dist-test-slave-zj8q"
server_key: "dafa5fc2e61e1abc0e3cc4ccb16bd580"
server_key_iv: "c0679d692f155b09f50a24d8541bd976"
server_key_version: "encryptionkey@0"
I20260806 02:52:40.690852 2815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/ts-2/wal/instance:
uuid: "a2b1f777a6334e5996f0fe527734080a"
format_stamp: "Formatted at 2026-08-06 02:52:40 on dist-test-slave-zj8q"
server_key: "dafa5fc2e61e1abc0e3cc4ccb16bd580"
server_key_iv: "c0679d692f155b09f50a24d8541bd976"
server_key_version: "encryptionkey@0"
I20260806 02:52:40.696382 2815 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 02:52:40.700197 2836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:40.701659 2815 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:52:40.701817 2815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/ts-2/wal
uuid: "a2b1f777a6334e5996f0fe527734080a"
format_stamp: "Formatted at 2026-08-06 02:52:40 on dist-test-slave-zj8q"
server_key: "dafa5fc2e61e1abc0e3cc4ccb16bd580"
server_key_iv: "c0679d692f155b09f50a24d8541bd976"
server_key_version: "encryptionkey@0"
I20260806 02:52:40.702057 2815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:52:40.720714 2609 heartbeater.cc:503] Master 127.25.211.190:39327 was elected leader, sending a full tablet report...
I20260806 02:52:40.723330 2815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:52:40.727815 2815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:52:40.728101 2815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:52:40.729090 2815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:52:40.730548 2815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:52:40.730624 2815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:40.730753 2815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:52:40.730806 2815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:40.763046 2815 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.131:35819
I20260806 02:52:40.763083 3008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.131:35819 every 8 connection(s)
I20260806 02:52:40.764359 2815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
I20260806 02:52:40.770344 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 2815
I20260806 02:52:40.770519 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/minicluster-data/ts-2/wal/instance
I20260806 02:52:40.770987 26446 external_mini_cluster.cc:1428] Setting key f0d075e8cc3430962416eee69b41ffaa
Aug 06 02:52:40 dist-test-slave-zj8q krb5kdc[2239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984760, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.131@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:52:40.785254 3011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:40.767111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:51723 (local address 127.25.211.190:39327)
0806 02:52:40.767573 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:40.767577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:40.768609 (+ 1032us) server_negotiation.cc:407] Connection header received
0806 02:52:40.770589 (+ 1980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:40.770595 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:40.770707 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:40.770818 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:40.772734 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:40.773614 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:40.775295 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:40.775685 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:40.775812 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:40.779464 (+ 3652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:40.779470 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:40.779473 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:40.779509 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:40.781906 (+ 2397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:40.782782 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:40.782786 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:40.782788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:40.782916 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:40.783367 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:40.783371 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:40.783373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:40.783657 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:40.783812 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:40.784565 (+ 753us) server_negotiation.cc:299] Negotiation successful
0806 02:52:40.784826 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":230,"threads_started":1}
I20260806 02:52:40.786666 3014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:40.767667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:39327 (local address 127.25.211.131:51723)
0806 02:52:40.768422 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:40.768473 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:40.770270 (+ 1797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:40.771110 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:40.771123 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:40.771837 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:40.772548 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:40.772572 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:40.773790 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:40.773794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:40.775144 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:40.775158 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:40.775929 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:40.775937 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:40.776175 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:40.776860 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:52:40.776890 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:40.779211 (+ 2321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:40.782062 (+ 2851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:40.782073 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:40.782090 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:40.782631 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:40.783029 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:40.783034 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:40.783038 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:40.783239 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:40.783791 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:40.783798 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:40.784171 (+ 373us) client_negotiation.cc:769] Sending connection context
0806 02:52:40.784439 (+ 268us) client_negotiation.cc:240] Negotiation successful
0806 02:52:40.785014 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":282,"threads_started":1}
I20260806 02:52:40.788184 3009 heartbeater.cc:347] Connected to a master server at 127.25.211.190:39327
I20260806 02:52:40.788635 3009 heartbeater.cc:464] Registering TS with master...
I20260806 02:52:40.789551 3009 heartbeater.cc:511] Master 127.25.211.190:39327 requested a full tablet report, sending...
I20260806 02:52:40.791330 2314 ts_manager.cc:194] Registered new tserver with Master: a2b1f777a6334e5996f0fe527734080a (127.25.211.131:35819)
I20260806 02:52:40.792173 2314 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.211.131@KRBTEST.COM'} at 127.25.211.131:51723
I20260806 02:52:40.799861 26446 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/krb5kdc/test-admin.keytab.
Aug 06 02:52:40 dist-test-slave-zj8q krb5kdc[2239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:52:40.831588 26446 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260806 02:52:40.836957 3022 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:52:40.834352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58732 (local address 127.25.211.129:40235)
0806 02:52:40.834982 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:40.834992 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:40.835023 (+ 31us) server_negotiation.cc:407] Connection header received
0806 02:52:40.835106 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:40.835113 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:40.835327 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:40.835501 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:40.836515 (+ 1014us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58732: BlockingRecv error: recv got EOF from 127.0.0.1:58732 (error 108)
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":254,"threads_started":1}
W20260806 02:52:40.837083 26446 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/krb5kdc/krb5cc' of type FILE
Aug 06 02:52:40 dist-test-slave-zj8q krb5kdc[2239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:52:40.838789 26446 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 02:52:40 dist-test-slave-zj8q krb5kdc[2239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:52:40.840189 26446 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 02:52:40 dist-test-slave-zj8q krb5kdc[2239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.129@KRBTEST.COM
I20260806 02:52:40.856770 3022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:40.841989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58744 (local address 127.25.211.129:40235)
0806 02:52:40.842221 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:40.842227 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:40.842378 (+ 151us) server_negotiation.cc:407] Connection header received
0806 02:52:40.842559 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:40.842564 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:40.842633 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:40.842720 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:40.843782 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:40.844810 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:40.846194 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:40.846800 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:40.846921 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:40.850530 (+ 3609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:40.850544 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:40.850561 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:40.850600 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:40.853605 (+ 3005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:40.854519 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:40.854528 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:40.854531 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:40.854680 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:40.855093 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:40.855097 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:40.855099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:40.855518 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:40.855826 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:40.856184 (+ 358us) server_negotiation.cc:299] Negotiation successful
0806 02:52:40.856465 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260806 02:52:40.863003 3022 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:52:40.860929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58756 (local address 127.25.211.129:40235)
0806 02:52:40.861100 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:40.861106 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:40.861501 (+ 395us) server_negotiation.cc:407] Connection header received
0806 02:52:40.861733 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:40.861739 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:40.861809 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:40.861891 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:40.862771 (+ 880us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58756: BlockingRecv error: recv got EOF from 127.0.0.1:58756 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260806 02:52:40.863567 26446 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/krb5kdc/krb5cc)
Aug 06 02:52:40 dist-test-slave-zj8q krb5kdc[2239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:52:40.865135 26446 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 02:52:40 dist-test-slave-zj8q krb5kdc[2239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:52:40.866489 26446 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 02:52:40 dist-test-slave-zj8q krb5kdc[2239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.129@KRBTEST.COM
I20260806 02:52:40.881793 3022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:40.868067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58770 (local address 127.25.211.129:40235)
0806 02:52:40.868315 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:40.868322 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:40.868517 (+ 195us) server_negotiation.cc:407] Connection header received
0806 02:52:40.868705 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:40.868710 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:40.868776 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:40.868863 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:40.869860 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:40.870645 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:40.871882 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:40.872337 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:40.872450 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:40.876276 (+ 3826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:40.876281 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:40.876285 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:40.876326 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:40.879096 (+ 2770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:40.879854 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:40.879859 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:40.879862 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:40.880005 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:40.880416 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:40.880420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:40.880422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:40.880729 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:40.880981 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:40.881287 (+ 306us) server_negotiation.cc:299] Negotiation successful
0806 02:52:40.881501 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
W20260806 02:52:40.886412 3022 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:52:40.884483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58780 (local address 127.25.211.129:40235)
0806 02:52:40.884634 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:40.884638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:40.885018 (+ 380us) server_negotiation.cc:407] Connection header received
0806 02:52:40.885197 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:40.885201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:40.885286 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:40.885368 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:40.886135 (+ 767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58780: BlockingRecv error: recv got EOF from 127.0.0.1:58780 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260806 02:52:40.886543 26446 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1785984707115702-26446-0/krb5kdc/krb5cc)
Aug 06 02:52:40 dist-test-slave-zj8q krb5kdc[2239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:52:40.888121 26446 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 02:52:40 dist-test-slave-zj8q krb5kdc[2239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:52:40.889456 26446 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 02:52:40 dist-test-slave-zj8q krb5kdc[2239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.129@KRBTEST.COM
I20260806 02:52:40.904242 3022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:40.891045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58790 (local address 127.25.211.129:40235)
0806 02:52:40.891263 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:40.891269 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:40.891419 (+ 150us) server_negotiation.cc:407] Connection header received
0806 02:52:40.891587 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:40.891591 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:40.891654 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:40.891745 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:40.892773 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:40.893513 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:40.894900 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:40.895322 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:40.895434 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:40.899009 (+ 3575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:40.899014 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:40.899017 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:40.899053 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:40.901553 (+ 2500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:40.902263 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:40.902269 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:40.902272 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:40.902454 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:40.902814 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:40.902818 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:40.902820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:40.903135 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:40.903445 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:40.903705 (+ 260us) server_negotiation.cc:299] Negotiation successful
0806 02:52:40.903936 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 02:52:40.906044 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 2414
I20260806 02:52:40.917666 3022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:40.906825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58798 (local address 127.25.211.129:40235)
0806 02:52:40.907010 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:40.907014 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:40.907345 (+ 331us) server_negotiation.cc:407] Connection header received
0806 02:52:40.907471 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:40.907474 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:40.907532 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:40.907609 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:40.908563 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:40.909282 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:40.910527 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:40.910981 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:40.911076 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:40.913184 (+ 2108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:40.913192 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:40.913195 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:40.913228 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:40.915649 (+ 2421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:40.916201 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:40.916204 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:40.916206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:40.916320 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:40.916595 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:40.916598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:40.916600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:40.916858 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:40.917050 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:40.917263 (+ 213us) server_negotiation.cc:299] Negotiation successful
0806 02:52:40.917420 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 02:52:40.918275 2602 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:52:41.004278 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 2414
I20260806 02:52:41.027848 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 2617
Aug 06 02:52:41 dist-test-slave-zj8q krb5kdc[2239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.130@KRBTEST.COM
I20260806 02:52:41.044288 3042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:41.028663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35176 (local address 127.25.211.130:40051)
0806 02:52:41.029269 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:41.029277 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:41.029304 (+ 27us) server_negotiation.cc:407] Connection header received
0806 02:52:41.029367 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:41.029373 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:41.029583 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:41.029720 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:41.030978 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:41.032109 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:41.033427 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:41.034030 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:41.034324 (+ 294us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:41.038304 (+ 3980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:41.038319 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:41.038335 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:41.038382 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:41.041202 (+ 2820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:41.042004 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:41.042011 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:41.042014 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:41.042197 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:41.042619 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:41.042622 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:41.042623 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:41.043015 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:41.043259 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:41.043649 (+ 390us) server_negotiation.cc:299] Negotiation successful
0806 02:52:41.043914 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":312,"threads_started":1}
I20260806 02:52:41.045724 2799 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:52:41.130096 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 2617
I20260806 02:52:41.151266 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 2815
Aug 06 02:52:41 dist-test-slave-zj8q krb5kdc[2239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.131@KRBTEST.COM
I20260806 02:52:41.168963 3045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:41.152396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35538 (local address 127.25.211.131:35819)
0806 02:52:41.153112 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:41.153119 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:41.153150 (+ 31us) server_negotiation.cc:407] Connection header received
0806 02:52:41.153219 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:41.153225 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:41.153445 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:41.153590 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:41.154891 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:41.156175 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:41.157889 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:41.158693 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:41.158812 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:41.163331 (+ 4519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:41.163348 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:41.163366 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:41.163414 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:41.166378 (+ 2964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:41.167073 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:41.167079 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:41.167081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:41.167217 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:41.167512 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:41.167515 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:41.167516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:41.167869 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:41.168207 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:41.168447 (+ 240us) server_negotiation.cc:299] Negotiation successful
0806 02:52:41.168668 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":393,"threads_started":1}
I20260806 02:52:41.170343 3001 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:52:41.254658 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 2815
I20260806 02:52:41.276731 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 2265
I20260806 02:52:41.277966 2366 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:52:41.382055 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 2265
2026-08-06T02:52:41Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4272 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:52:41 dist-test-slave-zj8q krb5kdc[3054](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:52:41 dist-test-slave-zj8q krb5kdc[3054](info): set up 2 sockets
Aug 06 02:52:41 dist-test-slave-zj8q krb5kdc[3054](info): commencing operation
krb5kdc: starting...
W20260806 02:52:42.590224 26446 mini_kdc.cc:121] Time spent starting KDC: real 1.159s user 0.004s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Aug 06 02:52:42 dist-test-slave-zj8q krb5kdc[3054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:52:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:52:42Z Disabled control of system clock
WARNING: no policy specified for oryx/127.25.211.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.211.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.211.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/krb5kdc/oryx_127.25.211.190.keytab.
Entry for principal oryx/127.25.211.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/krb5kdc/oryx_127.25.211.190.keytab.
I20260806 02:52:42.857544 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:45807
--webserver_interface=127.25.211.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/krb5kdc/oryx_127.25.211.190.keytab
--principal=oryx/127.25.211.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:40571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.211.190:45807
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:52:43.200731 3070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:52:43.201054 3070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:52:43.201159 3070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:52:43.208595 3070 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:52:43.208737 3070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:52:43.208793 3070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:52:43.208854 3070 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:52:43.208899 3070 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:52:43.219342 3070 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:40571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.211.190:45807
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/krb5kdc/oryx_127.25.211.190.keytab
--principal=oryx/127.25.211.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:45807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.211.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:52:43.220752 3070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:52:43.222126 3070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:52:43.234699 3076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:43.234717 3077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:43.234767 3080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:52:43.235284 3070 server_base.cc:1034] running on GCE node
I20260806 02:52:43.236212 3070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:52:43.237730 3070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:52:43.238972 3070 hybrid_clock.cc:648] HybridClock initialized: now 1785984763238924 us; error 33 us; skew 500 ppm
I20260806 02:52:43.239362 3070 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:52:43 dist-test-slave-zj8q krb5kdc[3054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984763, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.211.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:52:43.242671 3070 init.cc:377] Logged in from keytab as oryx/127.25.211.190@KRBTEST.COM (short username oryx)
I20260806 02:52:43.244462 3070 webserver.cc:460] Webserver started at http://127.25.211.190:43633/ using document root <none> and password file <none>
I20260806 02:52:43.245267 3070 fs_manager.cc:359] Metadata directory not provided
I20260806 02:52:43.245391 3070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:52:43.245751 3070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:52:43.249123 3070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-0/data/instance:
uuid: "5259274cdfee46ca8a88c8a0a0723c80"
format_stamp: "Formatted at 2026-08-06 02:52:43 on dist-test-slave-zj8q"
server_key: "41ff2d4e1f8035ad88b2baff73c61ec2"
server_key_iv: "17e2a6260c439a5ba21e4cdea1179274"
server_key_version: "encryptionkey@0"
I20260806 02:52:43.250027 3070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-0/wal/instance:
uuid: "5259274cdfee46ca8a88c8a0a0723c80"
format_stamp: "Formatted at 2026-08-06 02:52:43 on dist-test-slave-zj8q"
server_key: "41ff2d4e1f8035ad88b2baff73c61ec2"
server_key_iv: "17e2a6260c439a5ba21e4cdea1179274"
server_key_version: "encryptionkey@0"
I20260806 02:52:43.256093 3070 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 02:52:43.260119 3092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:43.261588 3070 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:52:43.261775 3070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-0/wal
uuid: "5259274cdfee46ca8a88c8a0a0723c80"
format_stamp: "Formatted at 2026-08-06 02:52:43 on dist-test-slave-zj8q"
server_key: "41ff2d4e1f8035ad88b2baff73c61ec2"
server_key_iv: "17e2a6260c439a5ba21e4cdea1179274"
server_key_version: "encryptionkey@0"
I20260806 02:52:43.261986 3070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:52:43.288386 3070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:52:43.292925 3070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:52:43.293229 3070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:52:43.311940 3070 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.190:45807
I20260806 02:52:43.311964 3178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.190:45807 every 8 connection(s)
I20260806 02:52:43.313383 3070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
I20260806 02:52:43.314919 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 3070
I20260806 02:52:43.315142 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-0/wal/instance
I20260806 02:52:43.316033 26446 external_mini_cluster.cc:1428] Setting key 6bd5076435aa1f87a29890d559ec34e8
I20260806 02:52:43.318604 3179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 06 02:52:43 dist-test-slave-zj8q krb5kdc[3054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM
I20260806 02:52:43.332645 3179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5259274cdfee46ca8a88c8a0a0723c80: Bootstrap starting.
I20260806 02:52:43.336285 3179 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5259274cdfee46ca8a88c8a0a0723c80: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:43.337699 3179 log.cc:824] T 00000000000000000000000000000000 P 5259274cdfee46ca8a88c8a0a0723c80: Log is configured to *not* fsync() on all Append() calls
I20260806 02:52:43.340386 3183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:43.318391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41160 (local address 127.25.211.190:45807)
0806 02:52:43.319235 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:43.319247 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:43.319290 (+ 43us) server_negotiation.cc:407] Connection header received
0806 02:52:43.320971 (+ 1681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:43.320995 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:43.321403 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:43.321778 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:43.323762 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:43.325208 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:43.327020 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:43.327747 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:43.327889 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:43.332863 (+ 4974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:43.332873 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:43.332890 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:43.332932 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:43.335927 (+ 2995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:43.336803 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:43.336811 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:43.336821 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:43.336992 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:43.337507 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:43.337512 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:43.337514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:43.338198 (+ 684us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:43.338534 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:43.339013 (+ 479us) server_negotiation.cc:299] Negotiation successful
0806 02:52:43.339393 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":401,"threads_started":1}
I20260806 02:52:43.340976 3179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5259274cdfee46ca8a88c8a0a0723c80: No bootstrap required, opened a new log
I20260806 02:52:43.349164 3179 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5259274cdfee46ca8a88c8a0a0723c80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5259274cdfee46ca8a88c8a0a0723c80" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45807 } }
I20260806 02:52:43.349604 3179 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5259274cdfee46ca8a88c8a0a0723c80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:43.349704 3179 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5259274cdfee46ca8a88c8a0a0723c80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5259274cdfee46ca8a88c8a0a0723c80, State: Initialized, Role: FOLLOWER
I20260806 02:52:43.350400 3179 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5259274cdfee46ca8a88c8a0a0723c80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5259274cdfee46ca8a88c8a0a0723c80" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45807 } }
I20260806 02:52:43.350637 3179 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5259274cdfee46ca8a88c8a0a0723c80 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:52:43.350777 3179 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5259274cdfee46ca8a88c8a0a0723c80 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:52:43.350940 3179 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5259274cdfee46ca8a88c8a0a0723c80 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:43.353302 3179 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5259274cdfee46ca8a88c8a0a0723c80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5259274cdfee46ca8a88c8a0a0723c80" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45807 } }
I20260806 02:52:43.353893 3179 leader_election.cc:302] T 00000000000000000000000000000000 P 5259274cdfee46ca8a88c8a0a0723c80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5259274cdfee46ca8a88c8a0a0723c80; no voters:
I20260806 02:52:43.354566 3179 leader_election.cc:288] T 00000000000000000000000000000000 P 5259274cdfee46ca8a88c8a0a0723c80 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:52:43.354700 3186 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5259274cdfee46ca8a88c8a0a0723c80 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:52:43.355357 3186 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5259274cdfee46ca8a88c8a0a0723c80 [term 1 LEADER]: Becoming Leader. State: Replica: 5259274cdfee46ca8a88c8a0a0723c80, State: Running, Role: LEADER
I20260806 02:52:43.355994 3186 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5259274cdfee46ca8a88c8a0a0723c80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5259274cdfee46ca8a88c8a0a0723c80" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45807 } }
I20260806 02:52:43.356662 3179 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5259274cdfee46ca8a88c8a0a0723c80 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:52:43.358852 3191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5259274cdfee46ca8a88c8a0a0723c80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5259274cdfee46ca8a88c8a0a0723c80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5259274cdfee46ca8a88c8a0a0723c80" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45807 } } }
I20260806 02:52:43.359225 3191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5259274cdfee46ca8a88c8a0a0723c80 [sys.catalog]: This master's current role is: LEADER
I20260806 02:52:43.362404 3192 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5259274cdfee46ca8a88c8a0a0723c80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5259274cdfee46ca8a88c8a0a0723c80. Latest consensus state: current_term: 1 leader_uuid: "5259274cdfee46ca8a88c8a0a0723c80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5259274cdfee46ca8a88c8a0a0723c80" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45807 } } }
I20260806 02:52:43.363054 3192 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5259274cdfee46ca8a88c8a0a0723c80 [sys.catalog]: This master's current role is: LEADER
I20260806 02:52:43.365156 3204 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:52:43.369843 3204 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:52:43.378286 3204 catalog_manager.cc:1347] Generated new cluster ID: c7f486c5c0544604a757ab99b279479f
I20260806 02:52:43.378423 3204 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:52:43.394065 3204 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:52:43.395483 3204 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:52:43.402527 3204 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5259274cdfee46ca8a88c8a0a0723c80: Generated new TSK 0
I20260806 02:52:43.403359 3204 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.25.211.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.211.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.211.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/krb5kdc/oryx_127.25.211.129.keytab.
Entry for principal oryx/127.25.211.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/krb5kdc/oryx_127.25.211.129.keytab.
I20260806 02:52:43.469228 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.129:0
--local_ip_for_outbound_sockets=127.25.211.129
--webserver_interface=127.25.211.129
--webserver_port=0
--tserver_master_addrs=127.25.211.190:45807
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/krb5kdc/oryx_127.25.211.129.keytab
--principal=oryx/127.25.211.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:40571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:52:43.815769 3224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:52:43.816097 3224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:52:43.816196 3224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:52:43.823659 3224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:52:43.823776 3224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:52:43.823932 3224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.129
I20260806 02:52:43.834470 3224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:40571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/krb5kdc/oryx_127.25.211.129.keytab
--principal=oryx/127.25.211.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.211.129
--webserver_port=0
--tserver_master_addrs=127.25.211.190:45807
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.129
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:52:43.835876 3224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:52:43.837271 3224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:52:43.851119 3229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:43.851205 3231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:43.851375 3234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:52:43.852069 3224 server_base.cc:1034] running on GCE node
I20260806 02:52:43.852731 3224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:52:43.853749 3224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:52:43.854972 3224 hybrid_clock.cc:648] HybridClock initialized: now 1785984763854894 us; error 69 us; skew 500 ppm
I20260806 02:52:43.855322 3224 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:52:43 dist-test-slave-zj8q krb5kdc[3054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984763, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.211.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:52:43.858775 3224 init.cc:377] Logged in from keytab as oryx/127.25.211.129@KRBTEST.COM (short username oryx)
I20260806 02:52:43.860325 3224 webserver.cc:460] Webserver started at http://127.25.211.129:46839/ using document root <none> and password file <none>
I20260806 02:52:43.861099 3224 fs_manager.cc:359] Metadata directory not provided
I20260806 02:52:43.861233 3224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:52:43.861552 3224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:52:43.865020 3224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-0/data/instance:
uuid: "9c31aad209d04d6eb97f15899d7972dd"
format_stamp: "Formatted at 2026-08-06 02:52:43 on dist-test-slave-zj8q"
server_key: "d02e2e2b252fcb291035bcb5cf0cbde1"
server_key_iv: "a634f0cebc954f69c9ceb1d15e1d37cb"
server_key_version: "encryptionkey@0"
I20260806 02:52:43.865949 3224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-0/wal/instance:
uuid: "9c31aad209d04d6eb97f15899d7972dd"
format_stamp: "Formatted at 2026-08-06 02:52:43 on dist-test-slave-zj8q"
server_key: "d02e2e2b252fcb291035bcb5cf0cbde1"
server_key_iv: "a634f0cebc954f69c9ceb1d15e1d37cb"
server_key_version: "encryptionkey@0"
I20260806 02:52:43.872262 3224 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.001s
I20260806 02:52:43.876102 3244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:43.877597 3224 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 02:52:43.877761 3224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-0/wal
uuid: "9c31aad209d04d6eb97f15899d7972dd"
format_stamp: "Formatted at 2026-08-06 02:52:43 on dist-test-slave-zj8q"
server_key: "d02e2e2b252fcb291035bcb5cf0cbde1"
server_key_iv: "a634f0cebc954f69c9ceb1d15e1d37cb"
server_key_version: "encryptionkey@0"
I20260806 02:52:43.877975 3224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:52:43.893010 3224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:52:43.897365 3224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:52:43.897652 3224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:52:43.898681 3224 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:52:43.900061 3224 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:52:43.900138 3224 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:43.900239 3224 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:52:43.900300 3224 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:43.931870 3224 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.129:44003
I20260806 02:52:43.931910 3410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.129:44003 every 8 connection(s)
I20260806 02:52:43.933252 3224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
I20260806 02:52:43.935935 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 3224
I20260806 02:52:43.936183 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-0/wal/instance
I20260806 02:52:43.937060 26446 external_mini_cluster.cc:1428] Setting key fa0404010f05e1033a1f969fe52697cb
Aug 06 02:52:43 dist-test-slave-zj8q krb5kdc[3054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984763, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.211.129@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM
I20260806 02:52:43.954102 3413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:43.935962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:33693 (local address 127.25.211.190:45807)
0806 02:52:43.938068 (+ 2106us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:43.938074 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:43.938092 (+ 18us) server_negotiation.cc:407] Connection header received
0806 02:52:43.939877 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:43.939883 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:43.939965 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:43.940085 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:43.941602 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:43.942578 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:43.944145 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:43.944664 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:43.944782 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:43.948295 (+ 3513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:43.948301 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:43.948306 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:43.948342 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:43.950912 (+ 2570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:43.951769 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:43.951773 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:43.951775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:43.951887 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:43.952338 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:43.952341 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:43.952343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:43.952585 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:43.952719 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:43.953655 (+ 936us) server_negotiation.cc:299] Negotiation successful
0806 02:52:43.953826 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1989,"thread_start_us":1873,"threads_started":1}
I20260806 02:52:43.955538 3414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:43.936431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45807 (local address 127.25.211.129:33693)
0806 02:52:43.937373 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:43.937422 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:43.939589 (+ 2167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:43.940276 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:43.940284 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:43.940807 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:43.941425 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:43.941444 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:43.942728 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:43.942732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:43.943994 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:43.944008 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:43.944860 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:43.944865 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:43.945068 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:43.945736 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:52:43.945769 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:43.948074 (+ 2305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:43.951070 (+ 2996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:43.951079 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:43.951095 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:43.951661 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:43.951991 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:43.951995 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:43.951998 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:43.952239 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:43.952727 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:43.952734 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:43.953166 (+ 432us) client_negotiation.cc:769] Sending connection context
0806 02:52:43.953542 (+ 376us) client_negotiation.cc:240] Negotiation successful
0806 02:52:43.954079 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":350,"threads_started":1}
I20260806 02:52:43.957162 3411 heartbeater.cc:347] Connected to a master server at 127.25.211.190:45807
I20260806 02:52:43.957463 3411 heartbeater.cc:464] Registering TS with master...
I20260806 02:52:43.958279 3411 heartbeater.cc:511] Master 127.25.211.190:45807 requested a full tablet report, sending...
I20260806 02:52:43.960436 3120 ts_manager.cc:194] Registered new tserver with Master: 9c31aad209d04d6eb97f15899d7972dd (127.25.211.129:44003)
I20260806 02:52:43.961866 3120 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.25.211.129@KRBTEST.COM'} at 127.25.211.129:33693
WARNING: no policy specified for oryx/127.25.211.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.211.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.211.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/krb5kdc/oryx_127.25.211.130.keytab.
Entry for principal oryx/127.25.211.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/krb5kdc/oryx_127.25.211.130.keytab.
I20260806 02:52:43.997352 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.130:0
--local_ip_for_outbound_sockets=127.25.211.130
--webserver_interface=127.25.211.130
--webserver_port=0
--tserver_master_addrs=127.25.211.190:45807
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/krb5kdc/oryx_127.25.211.130.keytab
--principal=oryx/127.25.211.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:40571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:52:44.364120 3418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:52:44.364452 3418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:52:44.364588 3418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:52:44.372129 3418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:52:44.372282 3418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:52:44.372442 3418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.130
I20260806 02:52:44.382604 3418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:40571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/krb5kdc/oryx_127.25.211.130.keytab
--principal=oryx/127.25.211.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.211.130
--webserver_port=0
--tserver_master_addrs=127.25.211.190:45807
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.130
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:52:44.384037 3418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:52:44.385430 3418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:52:44.399124 3433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:44.399080 3429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:44.399148 3428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:52:44.399922 3418 server_base.cc:1034] running on GCE node
I20260806 02:52:44.400679 3418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:52:44.401680 3418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:52:44.402894 3418 hybrid_clock.cc:648] HybridClock initialized: now 1785984764402836 us; error 50 us; skew 500 ppm
I20260806 02:52:44.403254 3418 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:52:44 dist-test-slave-zj8q krb5kdc[3054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984764, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.211.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:52:44.406622 3418 init.cc:377] Logged in from keytab as oryx/127.25.211.130@KRBTEST.COM (short username oryx)
I20260806 02:52:44.408207 3418 webserver.cc:460] Webserver started at http://127.25.211.130:33273/ using document root <none> and password file <none>
I20260806 02:52:44.408970 3418 fs_manager.cc:359] Metadata directory not provided
I20260806 02:52:44.409092 3418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:52:44.409412 3418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:52:44.412593 3418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-1/data/instance:
uuid: "916e85b805ce4cbab170fdf74b250f9e"
format_stamp: "Formatted at 2026-08-06 02:52:44 on dist-test-slave-zj8q"
server_key: "f29434c91229076829751b305685c169"
server_key_iv: "cbc9b745c10384d9ccda2ddf6d7ce657"
server_key_version: "encryptionkey@0"
I20260806 02:52:44.413471 3418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-1/wal/instance:
uuid: "916e85b805ce4cbab170fdf74b250f9e"
format_stamp: "Formatted at 2026-08-06 02:52:44 on dist-test-slave-zj8q"
server_key: "f29434c91229076829751b305685c169"
server_key_iv: "cbc9b745c10384d9ccda2ddf6d7ce657"
server_key_version: "encryptionkey@0"
I20260806 02:52:44.419165 3418 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260806 02:52:44.423028 3442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:44.424568 3418 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:52:44.424813 3418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-1/wal
uuid: "916e85b805ce4cbab170fdf74b250f9e"
format_stamp: "Formatted at 2026-08-06 02:52:44 on dist-test-slave-zj8q"
server_key: "f29434c91229076829751b305685c169"
server_key_iv: "cbc9b745c10384d9ccda2ddf6d7ce657"
server_key_version: "encryptionkey@0"
I20260806 02:52:44.425025 3418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:52:44.441978 3418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:52:44.446295 3418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:52:44.446571 3418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:52:44.447567 3418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:52:44.448997 3418 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:52:44.449075 3418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:44.449195 3418 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:52:44.449258 3418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:44.481654 3418 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.130:42313
I20260806 02:52:44.481685 3612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.130:42313 every 8 connection(s)
I20260806 02:52:44.483145 3418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
I20260806 02:52:44.487725 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 3418
I20260806 02:52:44.487910 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-1/wal/instance
I20260806 02:52:44.488474 26446 external_mini_cluster.cc:1428] Setting key d8be1ee338032d42035f311a7cafeb43
Aug 06 02:52:44 dist-test-slave-zj8q krb5kdc[3054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984764, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.211.130@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM
I20260806 02:52:44.506134 3617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:44.485757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:37469 (local address 127.25.211.190:45807)
0806 02:52:44.486248 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:44.486253 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:44.487157 (+ 904us) server_negotiation.cc:407] Connection header received
0806 02:52:44.489570 (+ 2413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:44.489576 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:44.489665 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:44.489764 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:44.491729 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:44.492483 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:44.494663 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:44.495229 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:44.495346 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:44.499633 (+ 4287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:44.499639 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:44.499642 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:44.499679 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:44.502810 (+ 3131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:44.503591 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:44.503597 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:44.503601 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:44.503778 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:44.504179 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:44.504183 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:44.504186 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:44.504575 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:44.504732 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:44.505458 (+ 726us) server_negotiation.cc:299] Negotiation successful
0806 02:52:44.505708 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":231,"threads_started":1}
I20260806 02:52:44.506992 3621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:44.486235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45807 (local address 127.25.211.130:37469)
0806 02:52:44.486969 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:44.487012 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:44.489295 (+ 2283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:44.490016 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:44.490025 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:44.490735 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:44.491531 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:44.491570 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:44.492653 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:44.492657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:44.494444 (+ 1787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:44.494459 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:44.495503 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:44.495510 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:44.495825 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:44.496566 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:52:44.496595 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:44.499346 (+ 2751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:44.502956 (+ 3610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:44.502964 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:44.502978 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:44.503441 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:44.503873 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:44.503876 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:44.503878 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:44.504070 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:44.504709 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:44.504715 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:44.505067 (+ 352us) client_negotiation.cc:769] Sending connection context
0806 02:52:44.505318 (+ 251us) client_negotiation.cc:240] Negotiation successful
0806 02:52:44.505766 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":287,"threads_started":1}
I20260806 02:52:44.508633 3613 heartbeater.cc:347] Connected to a master server at 127.25.211.190:45807
I20260806 02:52:44.508956 3613 heartbeater.cc:464] Registering TS with master...
I20260806 02:52:44.509778 3613 heartbeater.cc:511] Master 127.25.211.190:45807 requested a full tablet report, sending...
I20260806 02:52:44.511480 3120 ts_manager.cc:194] Registered new tserver with Master: 916e85b805ce4cbab170fdf74b250f9e (127.25.211.130:42313)
I20260806 02:52:44.512414 3120 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.25.211.130@KRBTEST.COM'} at 127.25.211.130:37469
WARNING: no policy specified for oryx/127.25.211.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.211.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.211.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/krb5kdc/oryx_127.25.211.131.keytab.
Entry for principal oryx/127.25.211.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/krb5kdc/oryx_127.25.211.131.keytab.
I20260806 02:52:44.547869 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.131:0
--local_ip_for_outbound_sockets=127.25.211.131
--webserver_interface=127.25.211.131
--webserver_port=0
--tserver_master_addrs=127.25.211.190:45807
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/krb5kdc/oryx_127.25.211.131.keytab
--principal=oryx/127.25.211.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:40571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:52:44.896445 3625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:52:44.896786 3625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:52:44.896943 3625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:52:44.904505 3625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:52:44.904645 3625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:52:44.904812 3625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.131
I20260806 02:52:44.914860 3625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:40571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/krb5kdc/oryx_127.25.211.131.keytab
--principal=oryx/127.25.211.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.211.131
--webserver_port=0
--tserver_master_addrs=127.25.211.190:45807
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.131
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:52:44.916235 3625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:52:44.917593 3625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:52:44.931183 3632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:44.931319 3635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:44.932243 3633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:52:44.932943 3625 server_base.cc:1034] running on GCE node
I20260806 02:52:44.933713 3625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:52:44.934983 3625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:52:44.936293 3625 hybrid_clock.cc:648] HybridClock initialized: now 1785984764936171 us; error 106 us; skew 500 ppm
I20260806 02:52:44.936692 3625 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:52:44 dist-test-slave-zj8q krb5kdc[3054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984764, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.211.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:52:44.940302 3625 init.cc:377] Logged in from keytab as oryx/127.25.211.131@KRBTEST.COM (short username oryx)
I20260806 02:52:44.941794 3625 webserver.cc:460] Webserver started at http://127.25.211.131:33695/ using document root <none> and password file <none>
I20260806 02:52:44.942610 3625 fs_manager.cc:359] Metadata directory not provided
I20260806 02:52:44.942736 3625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:52:44.943025 3625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:52:44.946189 3625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-2/data/instance:
uuid: "72d6e066b33b4d3089f69ac997a038e2"
format_stamp: "Formatted at 2026-08-06 02:52:44 on dist-test-slave-zj8q"
server_key: "a22224be660e706d6abfd65c5ceeb42c"
server_key_iv: "e6171d925ada3fd50936e7b7dc151c0e"
server_key_version: "encryptionkey@0"
I20260806 02:52:44.947119 3625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-2/wal/instance:
uuid: "72d6e066b33b4d3089f69ac997a038e2"
format_stamp: "Formatted at 2026-08-06 02:52:44 on dist-test-slave-zj8q"
server_key: "a22224be660e706d6abfd65c5ceeb42c"
server_key_iv: "e6171d925ada3fd50936e7b7dc151c0e"
server_key_version: "encryptionkey@0"
I20260806 02:52:44.952780 3625 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260806 02:52:44.956594 3647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:44.958271 3625 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:52:44.958438 3625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-2/wal
uuid: "72d6e066b33b4d3089f69ac997a038e2"
format_stamp: "Formatted at 2026-08-06 02:52:44 on dist-test-slave-zj8q"
server_key: "a22224be660e706d6abfd65c5ceeb42c"
server_key_iv: "e6171d925ada3fd50936e7b7dc151c0e"
server_key_version: "encryptionkey@0"
I20260806 02:52:44.958635 3625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:52:44.964968 3411 heartbeater.cc:503] Master 127.25.211.190:45807 was elected leader, sending a full tablet report...
I20260806 02:52:44.987068 3625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:52:44.992311 3625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:52:44.992591 3625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:52:44.993590 3625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:52:44.995321 3625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:52:44.995409 3625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:44.995498 3625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:52:44.995543 3625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:45.027613 3625 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.131:39997
I20260806 02:52:45.027650 3798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.131:39997 every 8 connection(s)
I20260806 02:52:45.028982 3625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
I20260806 02:52:45.035204 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 3625
I20260806 02:52:45.035408 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-2/wal/instance
I20260806 02:52:45.035975 26446 external_mini_cluster.cc:1428] Setting key 88080e944c245a474095fc7676c49e06
Aug 06 02:52:45 dist-test-slave-zj8q krb5kdc[3054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984764, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.211.131@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM
I20260806 02:52:45.052067 3801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:45.031822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:33531 (local address 127.25.211.190:45807)
0806 02:52:45.032322 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:45.032327 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:45.033413 (+ 1086us) server_negotiation.cc:407] Connection header received
0806 02:52:45.035696 (+ 2283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:45.035702 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:45.035792 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:45.035899 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:45.038069 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:45.039265 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:45.041018 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:45.041433 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:45.041533 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:45.045317 (+ 3784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:45.045323 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:45.045326 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:45.045365 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:45.048476 (+ 3111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:45.049326 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:45.049331 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:45.049333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:45.049498 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:45.050067 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:45.050073 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:45.050076 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:45.050451 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:45.050613 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:45.051414 (+ 801us) server_negotiation.cc:299] Negotiation successful
0806 02:52:45.051663 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":244,"threads_started":1}
I20260806 02:52:45.053083 3802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:45.032374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45807 (local address 127.25.211.131:33531)
0806 02:52:45.033227 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:45.033284 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:45.035353 (+ 2069us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:45.036224 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:45.036238 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:45.036952 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:45.037864 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:45.037894 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:45.039457 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:45.039462 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:45.040859 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:45.040879 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:45.041667 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:45.041672 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:45.041896 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:45.042605 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:52:45.042632 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:45.045070 (+ 2438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:45.048646 (+ 3576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:45.048655 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:45.048670 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:45.049177 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:45.049651 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:45.049655 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:45.049658 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:45.049917 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:45.050596 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:45.050605 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:45.051011 (+ 406us) client_negotiation.cc:769] Sending connection context
0806 02:52:45.051283 (+ 272us) client_negotiation.cc:240] Negotiation successful
0806 02:52:45.051888 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":354,"threads_started":1}
I20260806 02:52:45.054620 3799 heartbeater.cc:347] Connected to a master server at 127.25.211.190:45807
I20260806 02:52:45.054960 3799 heartbeater.cc:464] Registering TS with master...
I20260806 02:52:45.055720 3799 heartbeater.cc:511] Master 127.25.211.190:45807 requested a full tablet report, sending...
I20260806 02:52:45.057307 3120 ts_manager.cc:194] Registered new tserver with Master: 72d6e066b33b4d3089f69ac997a038e2 (127.25.211.131:39997)
I20260806 02:52:45.058173 3120 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.25.211.131@KRBTEST.COM'} at 127.25.211.131:33531
I20260806 02:52:45.064652 26446 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.25.211.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.211.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.211.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/krb5kdc/oryx_127.25.211.189.keytab.
Entry for principal oryx/127.25.211.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/krb5kdc/oryx_127.25.211.189.keytab.
I20260806 02:52:45.125865 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.189:44717
--webserver_interface=127.25.211.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/krb5kdc/oryx_127.25.211.189.keytab
--principal=oryx/127.25.211.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:40571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.211.190:45807,127.25.211.189:44717
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:52:45.485085 3809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:52:45.485484 3809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:52:45.485664 3809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:52:45.494006 3809 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:52:45.494133 3809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:52:45.494225 3809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:52:45.494289 3809 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:52:45.494335 3809 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:52:45.504549 3809 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:40571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.25.211.190:45807,127.25.211.189:44717
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/krb5kdc/oryx_127.25.211.189.keytab
--principal=oryx/127.25.211.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.189:44717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.25.211.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
W20260806 02:52:45.504936 3809 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.25.211.190:45807,127.25.211.189:44717'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260806 02:52:45.506029 3809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:52:45.507426 3809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 02:52:45.515232 3613 heartbeater.cc:503] Master 127.25.211.190:45807 was elected leader, sending a full tablet report...
W20260806 02:52:45.519352 3818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:45.519390 3817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:45.519419 3821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:52:45.520618 3809 server_base.cc:1034] running on GCE node
I20260806 02:52:45.521497 3809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:52:45.523052 3809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:52:45.524300 3809 hybrid_clock.cc:648] HybridClock initialized: now 1785984765524226 us; error 49 us; skew 500 ppm
I20260806 02:52:45.524638 3809 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:52:45 dist-test-slave-zj8q krb5kdc[3054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984765, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.211.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:52:45.528067 3809 init.cc:377] Logged in from keytab as oryx/127.25.211.189@KRBTEST.COM (short username oryx)
I20260806 02:52:45.529627 3809 webserver.cc:460] Webserver started at http://127.25.211.189:41711/ using document root <none> and password file <none>
I20260806 02:52:45.530439 3809 fs_manager.cc:359] Metadata directory not provided
I20260806 02:52:45.530548 3809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:52:45.530900 3809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:52:45.533991 3809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-1/data/instance:
uuid: "edad4ea4e4374f1383fb26cf1c316e03"
format_stamp: "Formatted at 2026-08-06 02:52:45 on dist-test-slave-zj8q"
server_key: "b8c04750fb199d158902bda540fdc7f2"
server_key_iv: "9f840cecededefe67da595d63e2391a8"
server_key_version: "encryptionkey@0"
I20260806 02:52:45.534857 3809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-1/wal/instance:
uuid: "edad4ea4e4374f1383fb26cf1c316e03"
format_stamp: "Formatted at 2026-08-06 02:52:45 on dist-test-slave-zj8q"
server_key: "b8c04750fb199d158902bda540fdc7f2"
server_key_iv: "9f840cecededefe67da595d63e2391a8"
server_key_version: "encryptionkey@0"
I20260806 02:52:45.540187 3809 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 02:52:45.543845 3831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:45.545194 3809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 02:52:45.545369 3809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-1/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-1/wal
uuid: "edad4ea4e4374f1383fb26cf1c316e03"
format_stamp: "Formatted at 2026-08-06 02:52:45 on dist-test-slave-zj8q"
server_key: "b8c04750fb199d158902bda540fdc7f2"
server_key_iv: "9f840cecededefe67da595d63e2391a8"
server_key_version: "encryptionkey@0"
I20260806 02:52:45.545594 3809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:52:45.562883 3809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:52:45.567735 3809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:52:45.568014 3809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:52:45.587065 3809 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.189:44717
I20260806 02:52:45.587085 3908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.189:44717 every 8 connection(s)
I20260806 02:52:45.588490 3809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-1/data/info.pb
I20260806 02:52:45.592343 3912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:45.593407 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 3809
I20260806 02:52:45.601495 3912 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45807 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:52:45 dist-test-slave-zj8q krb5kdc[3054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.25.211.189@KRBTEST.COM
Aug 06 02:52:45 dist-test-slave-zj8q krb5kdc[3054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984765, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.211.189@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM
I20260806 02:52:45.618423 3916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:45.595977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54048 (local address 127.25.211.189:44717)
0806 02:52:45.596811 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:45.596824 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:45.596872 (+ 48us) server_negotiation.cc:407] Connection header received
0806 02:52:45.598901 (+ 2029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:45.598928 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:45.599368 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:45.599776 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:45.601300 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:45.602739 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:45.604123 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:45.605231 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:45.605414 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:45.610120 (+ 4706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:45.610132 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:45.610176 (+ 44us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:45.610239 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:45.613639 (+ 3400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:45.614455 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:45.614464 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:45.614475 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:45.614657 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:45.615118 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:45.615123 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:45.615126 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:45.615876 (+ 750us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:45.616179 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:45.616566 (+ 387us) server_negotiation.cc:299] Negotiation successful
0806 02:52:45.617031 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":362,"threads_started":1}
I20260806 02:52:45.620306 3921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:45.604103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41174 (local address 127.25.211.190:45807)
0806 02:52:45.605405 (+ 1302us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:45.605411 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:45.605429 (+ 18us) server_negotiation.cc:407] Connection header received
0806 02:52:45.605487 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:45.605492 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:45.605596 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:45.605696 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:45.607273 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:45.608019 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:45.609687 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:45.610260 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:45.610395 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:45.614120 (+ 3725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:45.614125 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:45.614129 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:45.614210 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:45.617393 (+ 3183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:45.618093 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:45.618099 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:45.618102 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:45.618294 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:45.618703 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:45.618708 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:45.618712 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:45.619048 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:45.619178 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:45.619730 (+ 552us) server_negotiation.cc:299] Negotiation successful
0806 02:52:45.619945 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1181,"thread_start_us":1068,"threads_started":1}
I20260806 02:52:45.620378 3919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:45.603979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45807 (local address 127.0.0.1:41174)
0806 02:52:45.604741 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:45.604786 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:45.604946 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:45.605804 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:45.605811 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:45.606670 (+ 859us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:45.607091 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:45.607107 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:45.608175 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:45.608181 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:45.609533 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:45.609548 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:45.610441 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:45.610446 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:45.610599 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:45.611309 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:52:45.611338 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:45.613907 (+ 2569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:45.617518 (+ 3611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:45.617527 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:45.617532 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:45.617976 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:45.618394 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:45.618398 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:45.618400 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:45.618594 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:45.619203 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:45.619211 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:45.619397 (+ 186us) client_negotiation.cc:769] Sending connection context
0806 02:52:45.619587 (+ 190us) client_negotiation.cc:240] Negotiation successful
0806 02:52:45.619944 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":308,"threads_started":1}
I20260806 02:52:45.622220 3912 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.211.189" port: 44717 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:52:45 dist-test-slave-zj8q krb5kdc[3054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984765, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.211.189@KRBTEST.COM for oryx/127.25.211.189@KRBTEST.COM
I20260806 02:52:45.634467 3919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:45.622949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.189:44717 (local address 127.0.0.1:54050)
0806 02:52:45.623145 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:45.623163 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:45.623255 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:45.623501 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:45.623504 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:45.623804 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:45.623935 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:45.623943 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:45.624881 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:45.624886 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:45.625793 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:45.625806 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:45.626581 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:45.626586 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:45.626739 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:45.627251 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:52:45.627269 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:45.629304 (+ 2035us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:45.632108 (+ 2804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:45.632112 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:45.632115 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:45.632502 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:45.632827 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:45.632831 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:45.632833 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:45.632965 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:45.633558 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:45.633561 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:45.633702 (+ 141us) client_negotiation.cc:769] Sending connection context
0806 02:52:45.633815 (+ 113us) client_negotiation.cc:240] Negotiation successful
0806 02:52:45.634096 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260806 02:52:45.634613 3916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:45.623172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54050 (local address 127.25.211.189:44717)
0806 02:52:45.623301 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:45.623305 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:45.623318 (+ 13us) server_negotiation.cc:407] Connection header received
0806 02:52:45.623354 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:45.623357 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:45.623406 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:45.623477 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:45.624041 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:45.624731 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:45.625940 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:45.626411 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:45.626508 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:45.629493 (+ 2985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:45.629499 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:45.629503 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:45.629532 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:45.631965 (+ 2433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:45.632603 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:45.632606 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:45.632609 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:45.632742 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:45.633099 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:45.633103 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:45.633105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:45.633443 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:45.633612 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:45.633949 (+ 337us) server_negotiation.cc:299] Negotiation successful
0806 02:52:45.634329 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":111,"server-negotiator.queue_time_us":43}
I20260806 02:52:45.639562 3912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03: Bootstrap starting.
I20260806 02:52:45.643177 3912 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:45.644282 3912 log.cc:824] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03: Log is configured to *not* fsync() on all Append() calls
I20260806 02:52:45.647209 3912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03: No bootstrap required, opened a new log
I20260806 02:52:45.654820 3912 raft_consensus.cc:348] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5259274cdfee46ca8a88c8a0a0723c80" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45807 } } peers { permanent_uuid: "edad4ea4e4374f1383fb26cf1c316e03" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 44717 } }
I20260806 02:52:45.655216 3912 raft_consensus.cc:374] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:45.655320 3912 raft_consensus.cc:729] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edad4ea4e4374f1383fb26cf1c316e03, State: Initialized, Role: FOLLOWER
I20260806 02:52:45.655895 3912 consensus_queue.cc:260] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5259274cdfee46ca8a88c8a0a0723c80" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45807 } } peers { permanent_uuid: "edad4ea4e4374f1383fb26cf1c316e03" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 44717 } }
I20260806 02:52:45.656855 3925 sys_catalog.cc:455] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5259274cdfee46ca8a88c8a0a0723c80" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45807 } } peers { permanent_uuid: "edad4ea4e4374f1383fb26cf1c316e03" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 44717 } } }
I20260806 02:52:45.657187 3925 sys_catalog.cc:458] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:52:45.658043 3912 sys_catalog.cc:564] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03 [sys.catalog]: configured and running, proceeding with master startup.
W20260806 02:52:45.667265 3940 catalog_manager.cc:1558] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:52:45.667483 3940 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:52:45.669405 3809 master_runner.cc:428] Detected that this master edad4ea4e4374f1383fb26cf1c316e03 is joining an existing cluster
I20260806 02:52:45.669507 3809 master_runner.cc:432] Initiating AddMaster RPC to add 127.25.211.189:44717
I20260806 02:52:45.695688 3921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:45.674659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41182 (local address 127.25.211.190:45807)
0806 02:52:45.674836 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:45.674841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:45.678976 (+ 4135us) server_negotiation.cc:407] Connection header received
0806 02:52:45.679307 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:45.679314 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:45.679421 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:45.679499 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:45.681221 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:45.682364 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:45.684461 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:45.685135 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:45.685245 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:45.688462 (+ 3217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:45.688470 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:45.688475 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:45.688523 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:45.692158 (+ 3635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:45.693045 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:45.693052 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:45.693055 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:45.693226 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:45.693750 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:45.693756 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:45.693759 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:45.694146 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:45.694325 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:45.695035 (+ 710us) server_negotiation.cc:299] Negotiation successful
0806 02:52:45.695312 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 02:52:45.697227 3947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:45.674121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45807 (local address 127.0.0.1:41182)
0806 02:52:45.678829 (+ 4708us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:45.678860 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:45.679126 (+ 266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:45.679761 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:45.679766 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:45.680522 (+ 756us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:45.680965 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:45.680980 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:45.682593 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:45.682602 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:45.684276 (+ 1674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:45.684292 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:45.685607 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:45.685614 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:45.685949 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:45.686678 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:52:45.686703 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:45.688194 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:45.692334 (+ 4140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:45.692340 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:45.692344 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:45.692886 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:45.693384 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:45.693389 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:45.693393 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:45.693583 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:45.694434 (+ 851us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:45.694439 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:45.694673 (+ 234us) client_negotiation.cc:769] Sending connection context
0806 02:52:45.696293 (+ 1620us) client_negotiation.cc:240] Negotiation successful
0806 02:52:45.696734 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4537,"thread_start_us":4307,"threads_started":1}
I20260806 02:52:45.702527 3916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:45.679277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54054 (local address 127.25.211.189:44717)
0806 02:52:45.679453 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:45.679458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:45.686705 (+ 7247us) server_negotiation.cc:407] Connection header received
0806 02:52:45.687150 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:45.687155 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:45.687248 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:45.687333 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:45.688837 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:45.689835 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:45.691655 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:45.692328 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:45.692436 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:45.694785 (+ 2349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:45.694791 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:45.694795 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:45.694827 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:45.698720 (+ 3893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:45.699606 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:45.699612 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:45.699615 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:45.699802 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:45.700248 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:45.700253 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:45.700256 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:45.700660 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:45.700828 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:45.701854 (+ 1026us) server_negotiation.cc:299] Negotiation successful
0806 02:52:45.702115 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
WARNING: no policy specified for oryx/127.25.211.188@KRBTEST.COM; defaulting to no policy
I20260806 02:52:45.704974 3948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:45.679811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.189:44717 (local address 127.0.0.1:54054)
0806 02:52:45.686600 (+ 6789us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:45.686629 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:45.687021 (+ 392us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:45.687527 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:45.687531 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:45.688366 (+ 835us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:45.688690 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:45.688703 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:45.690053 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:45.690057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:45.691517 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:45.691532 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:45.692741 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:45.692746 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:45.692900 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:45.693504 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:52:45.693527 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:45.694601 (+ 1074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:45.698880 (+ 4279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:45.698884 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:45.698888 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:45.699478 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:45.699949 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:45.699954 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:45.699958 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:45.700129 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:45.700899 (+ 770us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:45.700903 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:45.701509 (+ 606us) client_negotiation.cc:769] Sending connection context
0806 02:52:45.703224 (+ 1715us) client_negotiation.cc:240] Negotiation successful
0806 02:52:45.704539 (+ 1315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6616,"thread_start_us":6885,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.211.188@KRBTEST.COM" created.
Aug 06 02:52:45 dist-test-slave-zj8q krb5kdc[3054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984763, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.211.190@KRBTEST.COM for oryx/127.25.211.189@KRBTEST.COM
I20260806 02:52:45.724908 3950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:45.707022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.189:44717 (local address 127.0.0.1:54068)
0806 02:52:45.707747 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:45.707797 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:45.708030 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:45.708536 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:45.708544 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:45.709404 (+ 860us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:45.709735 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:45.709749 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:45.711276 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:45.711281 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:45.712667 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:45.712685 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:45.714040 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:45.714047 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:45.714310 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:45.715072 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:52:45.715098 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:45.718062 (+ 2964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:45.721679 (+ 3617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:45.721693 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:45.721702 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:45.722346 (+ 644us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:45.722782 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:45.722787 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:45.722791 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:45.722974 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:45.723687 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:45.723695 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:45.723913 (+ 218us) client_negotiation.cc:769] Sending connection context
0806 02:52:45.724142 (+ 229us) client_negotiation.cc:240] Negotiation successful
0806 02:52:45.724498 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":311,"threads_started":1}
I20260806 02:52:45.725862 3916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:45.706922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54068 (local address 127.25.211.189:44717)
0806 02:52:45.707085 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:45.707092 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:45.707882 (+ 790us) server_negotiation.cc:407] Connection header received
0806 02:52:45.708261 (+ 379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:45.708267 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:45.708356 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:45.708473 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:45.709923 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:45.711101 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:45.712839 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:45.713612 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:45.713763 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:45.718298 (+ 4535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:45.718303 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:45.718306 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:45.718340 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:45.721511 (+ 3171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:45.722484 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:45.722489 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:45.722492 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:45.722654 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:45.723093 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:45.723098 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:45.723100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:45.723471 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:45.723595 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:45.725260 (+ 1665us) server_negotiation.cc:299] Negotiation successful
0806 02:52:45.725483 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 02:52:45.728066 3120 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.25.211.189:44717: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "edad4ea4e4374f1383fb26cf1c316e03"
member_type: NON_VOTER
last_known_addr {
host: "127.25.211.189"
port: 44717
}
attrs {
promote: true
}
}
dest_uuid: "5259274cdfee46ca8a88c8a0a0723c80"
cas_config_opid_index: -1
I20260806 02:52:45.729839 3120 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5259274cdfee46ca8a88c8a0a0723c80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5259274cdfee46ca8a88c8a0a0723c80" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45807 } } peers { permanent_uuid: "edad4ea4e4374f1383fb26cf1c316e03" member_type: NON_VOTER last_known_addr { host: "127.25.211.189" port: 44717 } attrs { promote: true } }
I20260806 02:52:45.730567 3925 raft_consensus.cc:482] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:52:45.730899 3925 raft_consensus.cc:504] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5259274cdfee46ca8a88c8a0a0723c80" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45807 } } peers { permanent_uuid: "edad4ea4e4374f1383fb26cf1c316e03" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 44717 } }
I20260806 02:52:45.732376 3925 leader_election.cc:288] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5259274cdfee46ca8a88c8a0a0723c80 (127.25.211.190:45807)
I20260806 02:52:45.733125 3139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "edad4ea4e4374f1383fb26cf1c316e03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5259274cdfee46ca8a88c8a0a0723c80" is_pre_election: true
I20260806 02:52:45.734337 3834 leader_election.cc:302] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: edad4ea4e4374f1383fb26cf1c316e03; no voters: 5259274cdfee46ca8a88c8a0a0723c80
I20260806 02:52:45.735037 3925 raft_consensus.cc:2726] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 02:52:45.739657 3956 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5259274cdfee46ca8a88c8a0a0723c80 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER edad4ea4e4374f1383fb26cf1c316e03 (127.25.211.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5259274cdfee46ca8a88c8a0a0723c80" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45807 } } peers { permanent_uuid: "edad4ea4e4374f1383fb26cf1c316e03" member_type: NON_VOTER last_known_addr { host: "127.25.211.189" port: 44717 } attrs { promote: true } } }
I20260806 02:52:45.740828 3873 raft_consensus.cc:3037] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:45.744335 3873 raft_consensus.cc:1261] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03 [term 1 FOLLOWER]: Refusing update from remote peer 5259274cdfee46ca8a88c8a0a0723c80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260806 02:52:45.744609 3956 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.25.211.189:44717
I20260806 02:52:45.745469 3956 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5259274cdfee46ca8a88c8a0a0723c80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edad4ea4e4374f1383fb26cf1c316e03" member_type: NON_VOTER last_known_addr { host: "127.25.211.189" port: 44717 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:52:45.745882 3809 master.cc:537] Master@127.25.211.189:44717 shutting down...
I20260806 02:52:45.746093 3955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5259274cdfee46ca8a88c8a0a0723c80 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5259274cdfee46ca8a88c8a0a0723c80" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5259274cdfee46ca8a88c8a0a0723c80" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45807 } } peers { permanent_uuid: "edad4ea4e4374f1383fb26cf1c316e03" member_type: NON_VOTER last_known_addr { host: "127.25.211.189" port: 44717 } attrs { promote: true } } }
I20260806 02:52:45.746484 3955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5259274cdfee46ca8a88c8a0a0723c80 [sys.catalog]: This master's current role is: LEADER
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.211.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/krb5kdc/oryx_127.25.211.188.keytab.
Entry for principal oryx/127.25.211.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/krb5kdc/oryx_127.25.211.188.keytab.
I20260806 02:52:45.749770 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.188:40381
--webserver_interface=127.25.211.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/krb5kdc/oryx_127.25.211.188.keytab
--principal=oryx/127.25.211.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:40571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.211.190:45807,127.25.211.189:44717,127.25.211.188:40381
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:52:45.760263 3098 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260806 02:52:45.760582 3809 raft_consensus.cc:2229] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:52:45.761173 3809 raft_consensus.cc:2258] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:52:45.761538 3809 tablet_replica.cc:330] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03: stopping tablet replica
I20260806 02:52:45.769882 3809 master.cc:552] Master@127.25.211.189:44717 shutdown complete.
I20260806 02:52:45.770113 3809 master_runner.cc:305] Clearing existing system tablet
I20260806 02:52:45.770730 3809 fs_manager.cc:359] Metadata directory not provided
I20260806 02:52:45.770906 3809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:52:45.771021 3950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:45.770521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.189:44717 (local address 127.0.0.1:54070)
0806 02:52:45.770703 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:45.770832 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.189:44717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260806 02:52:45.771371 3098 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5259274cdfee46ca8a88c8a0a0723c80 -> Peer edad4ea4e4374f1383fb26cf1c316e03 (127.25.211.189:44717): Couldn't send request to peer edad4ea4e4374f1383fb26cf1c316e03. Status: Network error: Client connection negotiation failed: client connection to 127.25.211.189:44717: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 02:52:45.774288 3809 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260806 02:52:45.776033 3964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:45.778560 3809 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:52:45.779160 3809 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260806 02:52:45.779343 3809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-1/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-1/wal
uuid: "edad4ea4e4374f1383fb26cf1c316e03"
format_stamp: "Formatted at 2026-08-06 02:52:45 on dist-test-slave-zj8q"
server_key: "b8c04750fb199d158902bda540fdc7f2"
server_key_iv: "9f840cecededefe67da595d63e2391a8"
server_key_version: "encryptionkey@0"
I20260806 02:52:45.785270 3809 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:52:45.793170 3809 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260806 02:52:45.793556 3809 log.cc:1196] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260806 02:52:45.794131 3809 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03: Deleting consensus metadata
I20260806 02:52:45.794507 3809 master_runner.cc:315] Copying system tablet from 127.25.211.190:45807
I20260806 02:52:45.797770 3809 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03: tablet copy: Beginning tablet copy session from remote peer at address 127.25.211.190:45807
I20260806 02:52:45.818188 3978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:45.798955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45807 (local address 127.0.0.1:41184)
0806 02:52:45.802684 (+ 3729us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:45.802713 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:45.802910 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:45.803353 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:45.803359 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:45.803961 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:45.804283 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:45.804299 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:45.805843 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:45.805850 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:45.807541 (+ 1691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:45.807560 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:45.808679 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:45.808685 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:45.808867 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:45.809583 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:52:45.809607 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:45.810787 (+ 1180us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:45.814831 (+ 4044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:45.814837 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:45.814841 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:45.815397 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:45.815860 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:45.815866 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:45.815869 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:45.816080 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:45.816760 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:45.816765 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:45.816970 (+ 205us) client_negotiation.cc:769] Sending connection context
0806 02:52:45.817150 (+ 180us) client_negotiation.cc:240] Negotiation successful
0806 02:52:45.817585 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3535,"thread_start_us":3388,"threads_started":1}
I20260806 02:52:45.818238 3921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:45.799027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41184 (local address 127.25.211.190:45807)
0806 02:52:45.799224 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:45.799231 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:45.802851 (+ 3620us) server_negotiation.cc:407] Connection header received
0806 02:52:45.803068 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:45.803074 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:45.803170 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:45.803290 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:45.804463 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:45.805643 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:45.807749 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:45.808465 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:45.808601 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:45.811028 (+ 2427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:45.811035 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:45.811040 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:45.811084 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:45.814633 (+ 3549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:45.815548 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:45.815554 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:45.815558 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:45.815737 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:45.816203 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:45.816209 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:45.816212 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:45.816587 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:45.816790 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:45.817301 (+ 511us) server_negotiation.cc:299] Negotiation successful
0806 02:52:45.817704 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 02:52:45.819736 3156 tablet_copy_service.cc:140] P 5259274cdfee46ca8a88c8a0a0723c80: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer edad4ea4e4374f1383fb26cf1c316e03 ({username='oryx', principal='oryx/127.25.211.189@KRBTEST.COM'} at 127.0.0.1:41184)
I20260806 02:52:45.820152 3156 tablet_copy_service.cc:161] P 5259274cdfee46ca8a88c8a0a0723c80: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer edad4ea4e4374f1383fb26cf1c316e03 at {username='oryx', principal='oryx/127.25.211.189@KRBTEST.COM'} at 127.0.0.1:41184: session id = edad4ea4e4374f1383fb26cf1c316e03-00000000000000000000000000000000
I20260806 02:52:45.825429 3156 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 5259274cdfee46ca8a88c8a0a0723c80: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 02:52:45.828912 3809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260806 02:52:45.836560 3809 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03: tablet copy: Starting download of 0 data blocks...
I20260806 02:52:45.837038 3809 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03: tablet copy: Starting download of 1 WAL segments...
I20260806 02:52:45.841526 3809 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 02:52:45.846843 3156 tablet_copy_service.cc:342] P 5259274cdfee46ca8a88c8a0a0723c80: Request end of tablet copy session edad4ea4e4374f1383fb26cf1c316e03-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.25.211.189@KRBTEST.COM'} at 127.0.0.1:41184
I20260806 02:52:45.847147 3156 tablet_copy_service.cc:433] P 5259274cdfee46ca8a88c8a0a0723c80: ending tablet copy session edad4ea4e4374f1383fb26cf1c316e03-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer edad4ea4e4374f1383fb26cf1c316e03
I20260806 02:52:45.858966 3809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:52:45.861527 3979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:45.862064 3982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:52:45.866825 3809 server_base.cc:1034] running on GCE node
W20260806 02:52:45.867197 3986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:52:45.867635 3809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:52:45.868274 3809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:52:45.869479 3809 hybrid_clock.cc:648] HybridClock initialized: now 1785984765869415 us; error 40 us; skew 500 ppm
I20260806 02:52:45.869695 3809 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:52:45 dist-test-slave-zj8q krb5kdc[3054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984765, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.211.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:52:45.871616 3809 init.cc:377] Logged in from keytab as oryx/127.25.211.189@KRBTEST.COM (short username oryx)
I20260806 02:52:45.872905 3809 webserver.cc:460] Webserver started at http://127.25.211.189:41235/ using document root <none> and password file <none>
I20260806 02:52:45.873271 3809 fs_manager.cc:359] Metadata directory not provided
I20260806 02:52:45.873363 3809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:52:45.875800 3809 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260806 02:52:45.877486 3997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:45.877887 3809 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 02:52:45.878005 3809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-1/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-1/wal
uuid: "edad4ea4e4374f1383fb26cf1c316e03"
format_stamp: "Formatted at 2026-08-06 02:52:45 on dist-test-slave-zj8q"
server_key: "b8c04750fb199d158902bda540fdc7f2"
server_key_iv: "9f840cecededefe67da595d63e2391a8"
server_key_version: "encryptionkey@0"
I20260806 02:52:45.878175 3809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:52:45.890640 3809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:52:45.893855 3809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:52:45.912106 3809 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.189:44717
I20260806 02:52:45.912137 4067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.189:44717 every 8 connection(s)
I20260806 02:52:45.913036 3809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-1/data/info.pb
I20260806 02:52:45.918969 4068 sys_catalog.cc:263] Verifying existing consensus state
I20260806 02:52:45.923858 4068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03: Bootstrap starting.
I20260806 02:52:45.940712 4068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:52:45.941262 4068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03: Bootstrap complete.
I20260806 02:52:45.942238 4068 raft_consensus.cc:348] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5259274cdfee46ca8a88c8a0a0723c80" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45807 } } peers { permanent_uuid: "edad4ea4e4374f1383fb26cf1c316e03" member_type: NON_VOTER last_known_addr { host: "127.25.211.189" port: 44717 } attrs { promote: true } }
I20260806 02:52:45.942469 4068 raft_consensus.cc:729] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: edad4ea4e4374f1383fb26cf1c316e03, State: Initialized, Role: LEARNER
I20260806 02:52:45.942700 4068 consensus_queue.cc:260] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5259274cdfee46ca8a88c8a0a0723c80" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45807 } } peers { permanent_uuid: "edad4ea4e4374f1383fb26cf1c316e03" member_type: NON_VOTER last_known_addr { host: "127.25.211.189" port: 44717 } attrs { promote: true } }
I20260806 02:52:45.943455 4071 sys_catalog.cc:455] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5259274cdfee46ca8a88c8a0a0723c80" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45807 } } peers { permanent_uuid: "edad4ea4e4374f1383fb26cf1c316e03" member_type: NON_VOTER last_known_addr { host: "127.25.211.189" port: 44717 } attrs { promote: true } } }
I20260806 02:52:45.943785 4068 sys_catalog.cc:564] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:52:45.943835 4071 sys_catalog.cc:458] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03 [sys.catalog]: This master's current role is: LEARNER
I20260806 02:52:45.951537 4093 catalog_manager.cc:1259] Loaded cluster ID: c7f486c5c0544604a757ab99b279479f
I20260806 02:52:45.951637 4093 catalog_manager.cc:1552] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03: loading cluster ID for follower catalog manager: success
I20260806 02:52:45.954124 4093 catalog_manager.cc:1574] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03: acquiring CA information for follower catalog manager: success
I20260806 02:52:45.956185 4093 catalog_manager.cc:1602] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 02:52:46.060985 3799 heartbeater.cc:503] Master 127.25.211.190:45807 was elected leader, sending a full tablet report...
I20260806 02:52:46.259266 4094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:46.251501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54076 (local address 127.25.211.189:44717)
0806 02:52:46.252187 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:46.252192 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:46.252213 (+ 21us) server_negotiation.cc:407] Connection header received
0806 02:52:46.252276 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:46.252280 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:46.252404 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:46.252512 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:52:46.253081 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:46.254592 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:46.257316 (+ 2724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:46.258227 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:46.258388 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:46.258528 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:46.258887 (+ 359us) server_negotiation.cc:299] Negotiation successful
0806 02:52:46.258991 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":397,"threads_started":1}
I20260806 02:52:46.259335 3950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:46.251256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.189:44717 (local address 127.0.0.1:54076)
0806 02:52:46.251426 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:46.251444 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:46.251547 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:46.252612 (+ 1065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:46.252619 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:46.252656 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:52:46.252878 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:46.252893 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:46.254810 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:46.254817 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:46.257080 (+ 2263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:46.257099 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:46.258420 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:46.258427 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:46.258677 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:46.258747 (+ 70us) client_negotiation.cc:769] Sending connection context
0806 02:52:46.258856 (+ 109us) client_negotiation.cc:240] Negotiation successful
0806 02:52:46.259059 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260806 02:52:46.261026 4033 raft_consensus.cc:1203] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260806 02:52:46.261662 4071 sys_catalog.cc:455] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5259274cdfee46ca8a88c8a0a0723c80. Latest consensus state: current_term: 1 leader_uuid: "5259274cdfee46ca8a88c8a0a0723c80" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5259274cdfee46ca8a88c8a0a0723c80" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45807 } } peers { permanent_uuid: "edad4ea4e4374f1383fb26cf1c316e03" member_type: NON_VOTER last_known_addr { host: "127.25.211.189" port: 44717 } attrs { promote: true } } }
I20260806 02:52:46.261926 4071 sys_catalog.cc:458] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03 [sys.catalog]: This master's current role is: LEARNER
W20260806 02:52:46.269492 3961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:52:46.269814 3961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:52:46.269914 3961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:52:46.278128 3961 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:52:46.278285 3961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:52:46.278342 3961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:52:46.278401 3961 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:52:46.278524 3961 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:52:46.290683 3961 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:40571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.25.211.190:45807,127.25.211.189:44717,127.25.211.188:40381
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/krb5kdc/oryx_127.25.211.188.keytab
--principal=oryx/127.25.211.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.188:40381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.25.211.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:52:46.292244 3961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:52:46.293757 3961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:52:46.309319 4101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:46.309379 4104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:46.310104 4100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:52:46.310982 3961 server_base.cc:1034] running on GCE node
I20260806 02:52:46.312206 3961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:52:46.313939 3961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:52:46.315199 3961 hybrid_clock.cc:648] HybridClock initialized: now 1785984766315113 us; error 51 us; skew 500 ppm
I20260806 02:52:46.315699 3961 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:52:46 dist-test-slave-zj8q krb5kdc[3054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984766, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.211.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:52:46.319553 3961 init.cc:377] Logged in from keytab as oryx/127.25.211.188@KRBTEST.COM (short username oryx)
I20260806 02:52:46.321478 3961 webserver.cc:460] Webserver started at http://127.25.211.188:43625/ using document root <none> and password file <none>
I20260806 02:52:46.322276 3961 fs_manager.cc:359] Metadata directory not provided
I20260806 02:52:46.322414 3961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:52:46.322741 3961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:52:46.327178 3961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-2/data/instance:
uuid: "f8a064e7081e4bc7b6da2b9c03c7e98c"
format_stamp: "Formatted at 2026-08-06 02:52:46 on dist-test-slave-zj8q"
server_key: "a452412306030dc9585d0bc6d76abb51"
server_key_iv: "f1f181d899ec2a11dba0fc43e2b2f7df"
server_key_version: "encryptionkey@0"
I20260806 02:52:46.328186 3961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-2/wal/instance:
uuid: "f8a064e7081e4bc7b6da2b9c03c7e98c"
format_stamp: "Formatted at 2026-08-06 02:52:46 on dist-test-slave-zj8q"
server_key: "a452412306030dc9585d0bc6d76abb51"
server_key_iv: "f1f181d899ec2a11dba0fc43e2b2f7df"
server_key_version: "encryptionkey@0"
I20260806 02:52:46.335039 3961 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260806 02:52:46.339624 4116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:46.341055 3961 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260806 02:52:46.341243 3961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-2/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-2/wal
uuid: "f8a064e7081e4bc7b6da2b9c03c7e98c"
format_stamp: "Formatted at 2026-08-06 02:52:46 on dist-test-slave-zj8q"
server_key: "a452412306030dc9585d0bc6d76abb51"
server_key_iv: "f1f181d899ec2a11dba0fc43e2b2f7df"
server_key_version: "encryptionkey@0"
I20260806 02:52:46.341447 3961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:52:46.361831 3961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:52:46.367496 3961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:52:46.367976 3961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:52:46.397841 3961 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.188:40381
I20260806 02:52:46.397917 4197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.188:40381 every 8 connection(s)
I20260806 02:52:46.399686 3961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-2/data/info.pb
I20260806 02:52:46.403882 4201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:46.409862 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 3961
I20260806 02:52:46.413462 4201 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45807 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:52:46 dist-test-slave-zj8q krb5kdc[3054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984766, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.211.188@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM
Aug 06 02:52:46 dist-test-slave-zj8q krb5kdc[3054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.25.211.188@KRBTEST.COM
I20260806 02:52:46.437578 4211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:46.418258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41188 (local address 127.25.211.190:45807)
0806 02:52:46.418971 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:46.418982 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:46.419741 (+ 759us) server_negotiation.cc:407] Connection header received
0806 02:52:46.419973 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:46.419977 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:46.420046 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:46.420134 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:46.422123 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:46.423397 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:46.425606 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:46.426142 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:46.426302 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:46.429975 (+ 3673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:46.429980 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:46.429983 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:46.430019 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:46.433871 (+ 3852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:46.434870 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:46.434876 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:46.434879 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:46.435080 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:46.435589 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:46.435595 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:46.435598 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:46.435990 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:46.436174 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:46.436908 (+ 734us) server_negotiation.cc:299] Negotiation successful
0806 02:52:46.437139 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":373,"threads_started":1}
I20260806 02:52:46.439041 4208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:46.412556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43472 (local address 127.25.211.188:40381)
0806 02:52:46.413575 (+ 1019us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:46.413588 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:46.413634 (+ 46us) server_negotiation.cc:407] Connection header received
0806 02:52:46.416159 (+ 2525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:46.416190 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:46.416702 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:46.417147 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:46.419108 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:46.421591 (+ 2483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:46.423576 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:46.425086 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:46.425301 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:46.430472 (+ 5171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:46.430486 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:46.430505 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:46.430551 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:46.433971 (+ 3420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:46.434819 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:46.434829 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:46.434832 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:46.435047 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:46.435585 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:46.435591 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:46.435594 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:46.436329 (+ 735us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:46.436668 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:46.437025 (+ 357us) server_negotiation.cc:299] Negotiation successful
0806 02:52:46.437625 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":402,"server-negotiator.queue_time_us":800,"thread_start_us":550,"threads_started":1}
I20260806 02:52:46.439208 4212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:46.418331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45807 (local address 127.0.0.1:41188)
0806 02:52:46.419538 (+ 1207us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:46.419590 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:46.419839 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:46.420239 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:46.420246 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:46.421361 (+ 1115us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:46.421951 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:46.421963 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:46.423576 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:46.423582 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:46.425421 (+ 1839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:46.425435 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:46.426399 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:46.426405 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:46.426537 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:46.427134 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:52:46.427156 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:46.429706 (+ 2550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:46.434030 (+ 4324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:46.434045 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:46.434058 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:46.434746 (+ 688us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:46.435231 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:46.435236 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:46.435239 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:46.435465 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:46.436158 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:46.436164 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:46.436495 (+ 331us) client_negotiation.cc:769] Sending connection context
0806 02:52:46.436795 (+ 300us) client_negotiation.cc:240] Negotiation successful
0806 02:52:46.437346 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":908,"mutex_wait_us":8,"thread_start_us":541,"threads_started":1}
I20260806 02:52:46.440691 4201 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.211.189" port: 44717 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:52:46 dist-test-slave-zj8q krb5kdc[3054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984766, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.211.188@KRBTEST.COM for oryx/127.25.211.189@KRBTEST.COM
I20260806 02:52:46.456275 4212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:46.441796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.189:44717 (local address 127.0.0.1:54086)
0806 02:52:46.442000 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:46.442019 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:46.442133 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:46.442546 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:46.442550 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:46.442949 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:46.443193 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:46.443203 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:46.444352 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:46.444356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:46.445541 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:46.445568 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:46.446493 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:46.446499 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:46.446813 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:46.447461 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:52:46.447479 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:46.449939 (+ 2460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:46.453748 (+ 3809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:46.453754 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:46.453767 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:46.454215 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:46.454599 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:46.454604 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:46.454608 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:46.454751 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:46.455287 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:46.455291 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:46.455440 (+ 149us) client_negotiation.cc:769] Sending connection context
0806 02:52:46.455561 (+ 121us) client_negotiation.cc:240] Negotiation successful
0806 02:52:46.455905 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260806 02:52:46.456455 4094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:46.441621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54086 (local address 127.25.211.189:44717)
0806 02:52:46.441799 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:46.441803 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:46.442101 (+ 298us) server_negotiation.cc:407] Connection header received
0806 02:52:46.442319 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:46.442325 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:46.442410 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:46.442494 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:46.443329 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:46.444234 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:46.445733 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:46.446338 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:46.446465 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:46.450220 (+ 3755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:46.450226 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:46.450230 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:46.450273 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:46.453586 (+ 3313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:46.454332 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:46.454338 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:46.454342 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:46.454486 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:46.454863 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:46.454869 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:46.454871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:46.455169 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:46.455315 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:46.455670 (+ 355us) server_negotiation.cc:299] Negotiation successful
0806 02:52:46.455921 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 02:52:46.457816 4201 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.211.188" port: 40381 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:52:46 dist-test-slave-zj8q krb5kdc[3054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984766, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.211.188@KRBTEST.COM for oryx/127.25.211.188@KRBTEST.COM
I20260806 02:52:46.472002 4212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:46.458627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:40381 (local address 127.0.0.1:43488)
0806 02:52:46.458781 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:46.458796 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:46.458894 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:46.459154 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:46.459158 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:46.459599 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:46.459743 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:46.459752 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:46.460703 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:46.460707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:46.461672 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:46.461686 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:46.462469 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:46.462474 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:46.462767 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:46.463360 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:52:46.463377 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:46.465958 (+ 2581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:46.469394 (+ 3436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:46.469399 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:46.469404 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:46.469836 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:46.470224 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:46.470229 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:46.470234 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:46.470421 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:46.470936 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:46.470941 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:46.471079 (+ 138us) client_negotiation.cc:769] Sending connection context
0806 02:52:46.471250 (+ 171us) client_negotiation.cc:240] Negotiation successful
0806 02:52:46.471537 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260806 02:52:46.471999 4208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:46.458793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43488 (local address 127.25.211.188:40381)
0806 02:52:46.458926 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:46.458932 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:46.458947 (+ 15us) server_negotiation.cc:407] Connection header received
0806 02:52:46.458989 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:46.458994 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:46.459044 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:46.459123 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:46.459882 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:46.460579 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:46.461880 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:46.462328 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:46.462445 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:46.466139 (+ 3694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:46.466144 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:46.466147 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:46.466210 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:46.469242 (+ 3032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:46.469953 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:46.469959 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:46.469962 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:46.470088 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:46.470524 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:46.470530 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:46.470533 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:46.470824 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:46.470957 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:46.471359 (+ 402us) server_negotiation.cc:299] Negotiation successful
0806 02:52:46.471685 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":116,"server-negotiator.queue_time_us":40}
I20260806 02:52:46.477326 4201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c: Bootstrap starting.
I20260806 02:52:46.481559 4201 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:46.482816 4201 log.cc:824] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c: Log is configured to *not* fsync() on all Append() calls
I20260806 02:52:46.486053 4201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c: No bootstrap required, opened a new log
I20260806 02:52:46.495441 4201 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5259274cdfee46ca8a88c8a0a0723c80" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45807 } } peers { permanent_uuid: "edad4ea4e4374f1383fb26cf1c316e03" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 44717 } } peers { permanent_uuid: "f8a064e7081e4bc7b6da2b9c03c7e98c" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 40381 } }
I20260806 02:52:46.495849 4201 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:46.495978 4201 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8a064e7081e4bc7b6da2b9c03c7e98c, State: Initialized, Role: FOLLOWER
I20260806 02:52:46.496582 4201 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5259274cdfee46ca8a88c8a0a0723c80" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45807 } } peers { permanent_uuid: "edad4ea4e4374f1383fb26cf1c316e03" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 44717 } } peers { permanent_uuid: "f8a064e7081e4bc7b6da2b9c03c7e98c" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 40381 } }
I20260806 02:52:46.497709 4214 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5259274cdfee46ca8a88c8a0a0723c80" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45807 } } peers { permanent_uuid: "edad4ea4e4374f1383fb26cf1c316e03" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 44717 } } peers { permanent_uuid: "f8a064e7081e4bc7b6da2b9c03c7e98c" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 40381 } } }
I20260806 02:52:46.498085 4214 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:52:46.498822 4201 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c [sys.catalog]: configured and running, proceeding with master startup.
W20260806 02:52:46.511147 4233 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:52:46.511425 4233 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:52:46.515146 3961 master_runner.cc:428] Detected that this master f8a064e7081e4bc7b6da2b9c03c7e98c is joining an existing cluster
I20260806 02:52:46.515291 3961 master_runner.cc:432] Initiating AddMaster RPC to add 127.25.211.188:40381
I20260806 02:52:46.542595 4243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:46.521680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:40381 (local address 127.0.0.1:43502)
0806 02:52:46.523564 (+ 1884us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:46.523588 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:46.523711 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:46.524104 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:46.524109 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:46.525666 (+ 1557us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:46.526251 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:46.526264 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:46.528447 (+ 2183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:46.528453 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:46.530544 (+ 2091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:46.530564 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:46.532224 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:46.532233 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:46.532442 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:46.533178 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:52:46.533202 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:46.534681 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:46.539302 (+ 4621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:46.539307 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:46.539311 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:46.539888 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:46.540322 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:46.540327 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:46.540330 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:46.540487 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:46.541229 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:46.541234 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:46.541493 (+ 259us) client_negotiation.cc:769] Sending connection context
0806 02:52:46.541650 (+ 157us) client_negotiation.cc:240] Negotiation successful
0806 02:52:46.541982 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1750,"mutex_wait_us":57,"spinlock_wait_cycles":2944,"thread_start_us":1565,"threads_started":1}
I20260806 02:52:46.542625 4211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:46.520829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41200 (local address 127.25.211.190:45807)
0806 02:52:46.521048 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:46.521055 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:46.521643 (+ 588us) server_negotiation.cc:407] Connection header received
0806 02:52:46.521928 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:46.521934 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:46.522149 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:46.522254 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:46.526104 (+ 3850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:46.527297 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:46.530135 (+ 2838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:46.531031 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:46.531139 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:46.534833 (+ 3694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:46.534841 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:46.534845 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:46.534890 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:46.538717 (+ 3827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:46.539627 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:46.539633 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:46.539636 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:46.539813 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:46.540307 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:46.540312 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:46.540315 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:46.540709 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:46.540843 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:46.541648 (+ 805us) server_negotiation.cc:299] Negotiation successful
0806 02:52:46.542095 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 02:52:46.542614 4094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:46.520505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54100 (local address 127.25.211.189:44717)
0806 02:52:46.520709 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:46.520715 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:46.525361 (+ 4646us) server_negotiation.cc:407] Connection header received
0806 02:52:46.525611 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:46.525617 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:46.525691 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:46.525785 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:46.527629 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:46.528700 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:46.531186 (+ 2486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:46.532123 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:46.532219 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:46.534837 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:46.534843 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:46.534848 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:46.534890 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:46.539040 (+ 4150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:46.540013 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:46.540018 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:46.540022 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:46.540172 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:46.540670 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:46.540675 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:46.540678 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:46.541005 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:46.541125 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:46.541955 (+ 830us) server_negotiation.cc:299] Negotiation successful
0806 02:52:46.542257 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 02:52:46.544390 4208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:46.521750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43502 (local address 127.25.211.188:40381)
0806 02:52:46.521949 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:46.521955 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:46.523681 (+ 1726us) server_negotiation.cc:407] Connection header received
0806 02:52:46.523835 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:46.523845 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:46.523922 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:46.524029 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:46.526381 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:46.528049 (+ 1668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:46.530694 (+ 2645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:46.531580 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:46.531752 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:46.534861 (+ 3109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:46.534867 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:46.534872 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:46.534906 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:46.539160 (+ 4254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:46.540037 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:46.540042 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:46.540045 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:46.540203 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:46.540627 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:46.540632 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:46.540635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:46.541028 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:46.541176 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:46.543106 (+ 1930us) server_negotiation.cc:299] Negotiation successful
0806 02:52:46.543768 (+ 662us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":390,"server-negotiator.queue_time_us":60}
I20260806 02:52:46.545442 4242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:46.520659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45807 (local address 127.0.0.1:41200)
0806 02:52:46.521523 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:46.521552 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:46.521771 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:46.522486 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:46.522490 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:46.525373 (+ 2883us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:46.525893 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:46.525910 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:46.527505 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:46.527510 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:46.529937 (+ 2427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:46.529954 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:46.531585 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:46.531594 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:46.531839 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:46.532662 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:52:46.532686 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:46.534570 (+ 1884us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:46.538901 (+ 4331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:46.538907 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:46.538912 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:46.539473 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:46.539968 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:46.539972 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:46.539976 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:46.540182 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:46.540950 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:46.540958 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:46.541296 (+ 338us) client_negotiation.cc:769] Sending connection context
0806 02:52:46.542845 (+ 1549us) client_negotiation.cc:240] Negotiation successful
0806 02:52:46.543237 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"mutex_wait_us":40,"thread_start_us":526,"threads_started":1}
I20260806 02:52:46.546536 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 3224
I20260806 02:52:46.546774 4245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:46.522594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.189:44717 (local address 127.0.0.1:54100)
0806 02:52:46.525229 (+ 2635us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:46.525247 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:46.525462 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:46.525818 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:46.525822 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:46.526916 (+ 1094us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:46.527424 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:46.527438 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:46.528890 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:46.528894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:46.531024 (+ 2130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:46.531038 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:46.532556 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:46.532568 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:46.532783 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:46.533427 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:52:46.533456 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:46.534651 (+ 1195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:46.539213 (+ 4562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:46.539218 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:46.539222 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:46.539879 (+ 657us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:46.540328 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:46.540332 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:46.540335 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:46.540529 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:46.541213 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:46.541219 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:46.541549 (+ 330us) client_negotiation.cc:769] Sending connection context
0806 02:52:46.543492 (+ 1943us) client_negotiation.cc:240] Negotiation successful
0806 02:52:46.543934 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2444,"mutex_wait_us":140,"spinlock_wait_cycles":44800,"thread_start_us":2165,"threads_started":1}
Aug 06 02:52:46 dist-test-slave-zj8q krb5kdc[3054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984763, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.211.190@KRBTEST.COM for oryx/127.25.211.188@KRBTEST.COM
Aug 06 02:52:46 dist-test-slave-zj8q krb5kdc[3054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.25.211.129@KRBTEST.COM
I20260806 02:52:46.567838 3950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:46.552135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:40381 (local address 127.0.0.1:43512)
0806 02:52:46.552339 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:46.552359 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:46.552477 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:46.552907 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:46.552912 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:46.553485 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:46.553725 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:46.553736 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:46.555027 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:46.555033 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:46.556414 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:46.556430 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:46.557400 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:46.557406 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:46.557569 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:46.558056 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:52:46.558090 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:46.560221 (+ 2131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:46.564695 (+ 4474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:46.564701 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:46.564705 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:46.565241 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:46.565728 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:46.565734 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:46.565738 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:46.565888 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:46.566697 (+ 809us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:46.566701 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:46.566889 (+ 188us) client_negotiation.cc:769] Sending connection context
0806 02:52:46.567032 (+ 143us) client_negotiation.cc:240] Negotiation successful
0806 02:52:46.567361 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260806 02:52:46.568109 4208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:46.552414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43512 (local address 127.25.211.188:40381)
0806 02:52:46.552572 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:46.552576 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:46.552592 (+ 16us) server_negotiation.cc:407] Connection header received
0806 02:52:46.552677 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:46.552680 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:46.552760 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:46.552850 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:46.553889 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:46.554868 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:46.556582 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:46.557241 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:46.557353 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:46.560449 (+ 3096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:46.560454 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:46.560458 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:46.560495 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:46.564548 (+ 4053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:46.565361 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:46.565367 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:46.565369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:46.565607 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:46.566018 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:46.566022 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:46.566024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:46.566489 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:46.566632 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:46.567228 (+ 596us) server_negotiation.cc:299] Negotiation successful
0806 02:52:46.567533 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 02:52:46.569996 3120 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.25.211.188:40381: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f8a064e7081e4bc7b6da2b9c03c7e98c"
member_type: NON_VOTER
last_known_addr {
host: "127.25.211.188"
port: 40381
}
attrs {
promote: true
}
}
dest_uuid: "5259274cdfee46ca8a88c8a0a0723c80"
cas_config_opid_index: 5
I20260806 02:52:46.570993 3120 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5259274cdfee46ca8a88c8a0a0723c80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5259274cdfee46ca8a88c8a0a0723c80" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45807 } } peers { permanent_uuid: "edad4ea4e4374f1383fb26cf1c316e03" member_type: NON_VOTER last_known_addr { host: "127.25.211.189" port: 44717 } attrs { promote: true } } peers { permanent_uuid: "f8a064e7081e4bc7b6da2b9c03c7e98c" member_type: NON_VOTER last_known_addr { host: "127.25.211.188" port: 40381 } attrs { promote: true } }
I20260806 02:52:46.572731 3956 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5259274cdfee46ca8a88c8a0a0723c80 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f8a064e7081e4bc7b6da2b9c03c7e98c (127.25.211.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5259274cdfee46ca8a88c8a0a0723c80" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45807 } } peers { permanent_uuid: "edad4ea4e4374f1383fb26cf1c316e03" member_type: NON_VOTER last_known_addr { host: "127.25.211.189" port: 44717 } attrs { promote: true } } peers { permanent_uuid: "f8a064e7081e4bc7b6da2b9c03c7e98c" member_type: NON_VOTER last_known_addr { host: "127.25.211.188" port: 40381 } attrs { promote: true } } }
I20260806 02:52:46.573794 4158 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:46.579838 4158 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c [term 1 FOLLOWER]: Refusing update from remote peer 5259274cdfee46ca8a88c8a0a0723c80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260806 02:52:46.582402 4248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:46.548426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39448 (local address 127.25.211.129:44003)
0806 02:52:46.549276 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:46.549285 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:46.549317 (+ 32us) server_negotiation.cc:407] Connection header received
0806 02:52:46.549392 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:46.549401 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:46.549733 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:46.549908 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:46.552283 (+ 2375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:46.554178 (+ 1895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:46.555913 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:46.556660 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:46.556788 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:46.564323 (+ 7535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:46.564349 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:46.564373 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:46.564426 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:46.569158 (+ 4732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:46.576344 (+ 7186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:46.576357 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:46.576361 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:46.576611 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:46.577829 (+ 1218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:46.577836 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:46.577839 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:46.578425 (+ 586us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:46.578810 (+ 385us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:46.581433 (+ 2623us) server_negotiation.cc:299] Negotiation successful
0806 02:52:46.581815 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":474,"threads_started":1}
I20260806 02:52:46.584419 3405 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:52:46.590747 4033 raft_consensus.cc:1261] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03 [term 1 LEARNER]: Refusing update from remote peer 5259274cdfee46ca8a88c8a0a0723c80: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260806 02:52:46.595005 3956 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.25.211.188:40381
I20260806 02:52:46.595286 4250 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5259274cdfee46ca8a88c8a0a0723c80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8a064e7081e4bc7b6da2b9c03c7e98c" member_type: NON_VOTER last_known_addr { host: "127.25.211.188" port: 40381 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 02:52:46.595880 3961 master.cc:537] Master@127.25.211.188:40381 shutting down...
I20260806 02:52:46.596750 4249 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5259274cdfee46ca8a88c8a0a0723c80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edad4ea4e4374f1383fb26cf1c316e03" member_type: NON_VOTER last_known_addr { host: "127.25.211.189" port: 44717 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 02:52:46.610674 4033 raft_consensus.cc:2932] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f8a064e7081e4bc7b6da2b9c03c7e98c (127.25.211.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5259274cdfee46ca8a88c8a0a0723c80" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45807 } } peers { permanent_uuid: "edad4ea4e4374f1383fb26cf1c316e03" member_type: NON_VOTER last_known_addr { host: "127.25.211.189" port: 44717 } attrs { promote: true } } peers { permanent_uuid: "f8a064e7081e4bc7b6da2b9c03c7e98c" member_type: NON_VOTER last_known_addr { host: "127.25.211.188" port: 40381 } attrs { promote: true } } }
I20260806 02:52:46.615108 4252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5259274cdfee46ca8a88c8a0a0723c80 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5259274cdfee46ca8a88c8a0a0723c80" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5259274cdfee46ca8a88c8a0a0723c80" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45807 } } peers { permanent_uuid: "edad4ea4e4374f1383fb26cf1c316e03" member_type: NON_VOTER last_known_addr { host: "127.25.211.189" port: 44717 } attrs { promote: true } } peers { permanent_uuid: "f8a064e7081e4bc7b6da2b9c03c7e98c" member_type: NON_VOTER last_known_addr { host: "127.25.211.188" port: 40381 } attrs { promote: true } } }
I20260806 02:52:46.615360 4071 sys_catalog.cc:455] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5259274cdfee46ca8a88c8a0a0723c80" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5259274cdfee46ca8a88c8a0a0723c80" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45807 } } peers { permanent_uuid: "edad4ea4e4374f1383fb26cf1c316e03" member_type: NON_VOTER last_known_addr { host: "127.25.211.189" port: 44717 } attrs { promote: true } } peers { permanent_uuid: "f8a064e7081e4bc7b6da2b9c03c7e98c" member_type: NON_VOTER last_known_addr { host: "127.25.211.188" port: 40381 } attrs { promote: true } } }
I20260806 02:52:46.615754 4252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5259274cdfee46ca8a88c8a0a0723c80 [sys.catalog]: This master's current role is: LEADER
I20260806 02:52:46.615752 4071 sys_catalog.cc:458] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03 [sys.catalog]: This master's current role is: LEARNER
W20260806 02:52:46.616456 3093 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5259274cdfee46ca8a88c8a0a0723c80 -> Peer f8a064e7081e4bc7b6da2b9c03c7e98c (127.25.211.188:40381): Couldn't send request to peer f8a064e7081e4bc7b6da2b9c03c7e98c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260806 02:52:46.618302 3956 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 5259274cdfee46ca8a88c8a0a0723c80: attempting to promote NON_VOTER edad4ea4e4374f1383fb26cf1c316e03 to VOTER
I20260806 02:52:46.619951 3956 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5259274cdfee46ca8a88c8a0a0723c80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5259274cdfee46ca8a88c8a0a0723c80" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45807 } } peers { permanent_uuid: "edad4ea4e4374f1383fb26cf1c316e03" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 44717 } attrs { promote: false } } peers { permanent_uuid: "f8a064e7081e4bc7b6da2b9c03c7e98c" member_type: NON_VOTER last_known_addr { host: "127.25.211.188" port: 40381 } attrs { promote: true } }
I20260806 02:52:46.623342 4033 raft_consensus.cc:1261] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03 [term 1 LEARNER]: Refusing update from remote peer 5259274cdfee46ca8a88c8a0a0723c80: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260806 02:52:46.624017 4252 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5259274cdfee46ca8a88c8a0a0723c80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edad4ea4e4374f1383fb26cf1c316e03" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 44717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260806 02:52:46.624496 3093 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5259274cdfee46ca8a88c8a0a0723c80 -> Peer f8a064e7081e4bc7b6da2b9c03c7e98c (127.25.211.188:40381): Couldn't send request to peer f8a064e7081e4bc7b6da2b9c03c7e98c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260806 02:52:46.626610 3956 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5259274cdfee46ca8a88c8a0a0723c80 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, edad4ea4e4374f1383fb26cf1c316e03 (127.25.211.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5259274cdfee46ca8a88c8a0a0723c80" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45807 } } peers { permanent_uuid: "edad4ea4e4374f1383fb26cf1c316e03" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 44717 } attrs { promote: false } } peers { permanent_uuid: "f8a064e7081e4bc7b6da2b9c03c7e98c" member_type: NON_VOTER last_known_addr { host: "127.25.211.188" port: 40381 } attrs { promote: true } } }
I20260806 02:52:46.627712 4033 raft_consensus.cc:2932] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, edad4ea4e4374f1383fb26cf1c316e03 (127.25.211.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5259274cdfee46ca8a88c8a0a0723c80" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45807 } } peers { permanent_uuid: "edad4ea4e4374f1383fb26cf1c316e03" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 44717 } attrs { promote: false } } peers { permanent_uuid: "f8a064e7081e4bc7b6da2b9c03c7e98c" member_type: NON_VOTER last_known_addr { host: "127.25.211.188" port: 40381 } attrs { promote: true } } }
W20260806 02:52:46.631318 4125 tablet.cc:2369] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:52:46.632268 4071 sys_catalog.cc:455] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5259274cdfee46ca8a88c8a0a0723c80" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5259274cdfee46ca8a88c8a0a0723c80" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45807 } } peers { permanent_uuid: "edad4ea4e4374f1383fb26cf1c316e03" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 44717 } attrs { promote: false } } peers { permanent_uuid: "f8a064e7081e4bc7b6da2b9c03c7e98c" member_type: NON_VOTER last_known_addr { host: "127.25.211.188" port: 40381 } attrs { promote: true } } }
I20260806 02:52:46.632525 4071 sys_catalog.cc:458] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:52:46.634473 3961 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:52:46.634913 3961 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:52:46.635113 3961 tablet_replica.cc:330] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c: stopping tablet replica
I20260806 02:52:46.639319 4255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5259274cdfee46ca8a88c8a0a0723c80 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5259274cdfee46ca8a88c8a0a0723c80" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5259274cdfee46ca8a88c8a0a0723c80" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45807 } } peers { permanent_uuid: "edad4ea4e4374f1383fb26cf1c316e03" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 44717 } attrs { promote: false } } peers { permanent_uuid: "f8a064e7081e4bc7b6da2b9c03c7e98c" member_type: NON_VOTER last_known_addr { host: "127.25.211.188" port: 40381 } attrs { promote: true } } }
I20260806 02:52:46.639784 4255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5259274cdfee46ca8a88c8a0a0723c80 [sys.catalog]: This master's current role is: LEADER
I20260806 02:52:46.640290 3956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5259274cdfee46ca8a88c8a0a0723c80 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5259274cdfee46ca8a88c8a0a0723c80" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5259274cdfee46ca8a88c8a0a0723c80" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45807 } } peers { permanent_uuid: "edad4ea4e4374f1383fb26cf1c316e03" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 44717 } attrs { promote: false } } peers { permanent_uuid: "f8a064e7081e4bc7b6da2b9c03c7e98c" member_type: NON_VOTER last_known_addr { host: "127.25.211.188" port: 40381 } attrs { promote: true } } }
I20260806 02:52:46.640574 3956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5259274cdfee46ca8a88c8a0a0723c80 [sys.catalog]: This master's current role is: LEADER
I20260806 02:52:46.642390 3961 master.cc:552] Master@127.25.211.188:40381 shutdown complete.
I20260806 02:52:46.642472 3961 master_runner.cc:305] Clearing existing system tablet
I20260806 02:52:46.642832 3961 fs_manager.cc:359] Metadata directory not provided
I20260806 02:52:46.642970 3961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:52:46.646786 3961 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260806 02:52:46.648301 4260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:46.648696 3961 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:52:46.648953 3961 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260806 02:52:46.649055 3961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-2/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-2/wal
uuid: "f8a064e7081e4bc7b6da2b9c03c7e98c"
format_stamp: "Formatted at 2026-08-06 02:52:46 on dist-test-slave-zj8q"
server_key: "a452412306030dc9585d0bc6d76abb51"
server_key_iv: "f1f181d899ec2a11dba0fc43e2b2f7df"
server_key_version: "encryptionkey@0"
I20260806 02:52:46.652925 3961 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:52:46.658442 3961 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260806 02:52:46.658634 3961 log.cc:1196] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260806 02:52:46.659071 3961 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c: Deleting consensus metadata
I20260806 02:52:46.659330 3961 master_runner.cc:315] Copying system tablet from 127.25.211.190:45807
I20260806 02:52:46.662427 3961 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c: tablet copy: Beginning tablet copy session from remote peer at address 127.25.211.190:45807
I20260806 02:52:46.677201 4211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:46.663333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41212 (local address 127.25.211.190:45807)
0806 02:52:46.663611 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:46.663618 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:46.664219 (+ 601us) server_negotiation.cc:407] Connection header received
0806 02:52:46.664427 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:46.664433 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:46.664522 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:46.664609 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:46.666266 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:46.667152 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:46.668625 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:46.669080 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:46.669213 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:46.671357 (+ 2144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:46.671363 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:46.671366 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:46.671400 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:46.674074 (+ 2674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:46.674851 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:46.674856 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:46.674862 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:46.674997 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:46.675474 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:46.675478 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:46.675480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:46.675774 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:46.675913 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:46.676336 (+ 423us) server_negotiation.cc:299] Negotiation successful
0806 02:52:46.676593 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260806 02:52:46.677511 4272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:46.663236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45807 (local address 127.0.0.1:41212)
0806 02:52:46.664107 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:46.664124 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:46.664292 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:46.664668 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:46.664673 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:46.665417 (+ 744us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:46.665992 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:46.666011 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:46.667300 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:46.667306 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:46.668477 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:46.668492 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:46.669242 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:46.669247 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:46.669458 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:46.670087 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:52:46.670105 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:46.671129 (+ 1024us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:46.674239 (+ 3110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:46.674244 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:46.674247 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:46.674732 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:46.675107 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:46.675112 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:46.675116 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:46.675360 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:46.675896 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:46.675900 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:46.676069 (+ 169us) client_negotiation.cc:769] Sending connection context
0806 02:52:46.676195 (+ 126us) client_negotiation.cc:240] Negotiation successful
0806 02:52:46.676718 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":737,"thread_start_us":462,"threads_started":1}
I20260806 02:52:46.678249 3156 tablet_copy_service.cc:140] P 5259274cdfee46ca8a88c8a0a0723c80: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f8a064e7081e4bc7b6da2b9c03c7e98c ({username='oryx', principal='oryx/127.25.211.188@KRBTEST.COM'} at 127.0.0.1:41212)
I20260806 02:52:46.678436 3156 tablet_copy_service.cc:161] P 5259274cdfee46ca8a88c8a0a0723c80: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f8a064e7081e4bc7b6da2b9c03c7e98c at {username='oryx', principal='oryx/127.25.211.188@KRBTEST.COM'} at 127.0.0.1:41212: session id = f8a064e7081e4bc7b6da2b9c03c7e98c-00000000000000000000000000000000
I20260806 02:52:46.680895 3156 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 5259274cdfee46ca8a88c8a0a0723c80: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 02:52:46.682766 3961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260806 02:52:46.688212 3961 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c: tablet copy: Starting download of 0 data blocks...
I20260806 02:52:46.688521 3961 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c: tablet copy: Starting download of 1 WAL segments...
I20260806 02:52:46.691071 3961 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 02:52:46.694970 3156 tablet_copy_service.cc:342] P 5259274cdfee46ca8a88c8a0a0723c80: Request end of tablet copy session f8a064e7081e4bc7b6da2b9c03c7e98c-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.25.211.188@KRBTEST.COM'} at 127.0.0.1:41212
I20260806 02:52:46.695109 3156 tablet_copy_service.cc:433] P 5259274cdfee46ca8a88c8a0a0723c80: ending tablet copy session f8a064e7081e4bc7b6da2b9c03c7e98c-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f8a064e7081e4bc7b6da2b9c03c7e98c
I20260806 02:52:46.704891 3961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:52:46.707453 4273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:46.711742 4275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:52:46.716269 3961 server_base.cc:1034] running on GCE node
W20260806 02:52:46.716601 4278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:52:46.717093 3961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:52:46.717753 3961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:52:46.718927 3961 hybrid_clock.cc:648] HybridClock initialized: now 1785984766718870 us; error 51 us; skew 500 ppm
I20260806 02:52:46.719153 3961 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:52:46 dist-test-slave-zj8q krb5kdc[3054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984766, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.211.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:52:46.721176 3961 init.cc:377] Logged in from keytab as oryx/127.25.211.188@KRBTEST.COM (short username oryx)
I20260806 02:52:46.727453 3961 webserver.cc:460] Webserver started at http://127.25.211.188:43887/ using document root <none> and password file <none>
I20260806 02:52:46.727792 3961 fs_manager.cc:359] Metadata directory not provided
I20260806 02:52:46.727891 3961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:52:46.731323 3961 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:52:46.733484 4286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:46.733886 3961 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260806 02:52:46.734009 3961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-2/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-2/wal
uuid: "f8a064e7081e4bc7b6da2b9c03c7e98c"
format_stamp: "Formatted at 2026-08-06 02:52:46 on dist-test-slave-zj8q"
server_key: "a452412306030dc9585d0bc6d76abb51"
server_key_iv: "f1f181d899ec2a11dba0fc43e2b2f7df"
server_key_version: "encryptionkey@0"
I20260806 02:52:46.734222 3961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:52:46.743464 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 3224
I20260806 02:52:46.750269 3961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:52:46.754642 3961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:52:46.775599 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.129:44003
--local_ip_for_outbound_sockets=127.25.211.129
--tserver_master_addrs=127.25.211.190:45807
--webserver_port=46839
--webserver_interface=127.25.211.129
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/krb5kdc/oryx_127.25.211.129.keytab
--principal=oryx/127.25.211.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:40571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.25.211.190:45807,127.25.211.189:44717
--tserver_master_addrs=127.25.211.190:45807,127.25.211.189:44717,127.25.211.188:40381 with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:52:46.787051 3961 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.188:40381
I20260806 02:52:46.787106 4368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.188:40381 every 8 connection(s)
I20260806 02:52:46.788292 3961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/master-2/data/info.pb
I20260806 02:52:46.793422 4370 sys_catalog.cc:263] Verifying existing consensus state
I20260806 02:52:46.797171 4370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c: Bootstrap starting.
I20260806 02:52:46.819540 4370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:52:46.820082 4370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c: Bootstrap complete.
I20260806 02:52:46.821174 4370 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5259274cdfee46ca8a88c8a0a0723c80" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45807 } } peers { permanent_uuid: "edad4ea4e4374f1383fb26cf1c316e03" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 44717 } attrs { promote: false } } peers { permanent_uuid: "f8a064e7081e4bc7b6da2b9c03c7e98c" member_type: NON_VOTER last_known_addr { host: "127.25.211.188" port: 40381 } attrs { promote: true } }
I20260806 02:52:46.821475 4370 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f8a064e7081e4bc7b6da2b9c03c7e98c, State: Initialized, Role: LEARNER
I20260806 02:52:46.821734 4370 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5259274cdfee46ca8a88c8a0a0723c80" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45807 } } peers { permanent_uuid: "edad4ea4e4374f1383fb26cf1c316e03" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 44717 } attrs { promote: false } } peers { permanent_uuid: "f8a064e7081e4bc7b6da2b9c03c7e98c" member_type: NON_VOTER last_known_addr { host: "127.25.211.188" port: 40381 } attrs { promote: true } }
I20260806 02:52:46.822963 4374 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5259274cdfee46ca8a88c8a0a0723c80" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45807 } } peers { permanent_uuid: "edad4ea4e4374f1383fb26cf1c316e03" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 44717 } attrs { promote: false } } peers { permanent_uuid: "f8a064e7081e4bc7b6da2b9c03c7e98c" member_type: NON_VOTER last_known_addr { host: "127.25.211.188" port: 40381 } attrs { promote: true } } }
I20260806 02:52:46.823302 4370 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:52:46.823451 4374 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c [sys.catalog]: This master's current role is: LEARNER
I20260806 02:52:46.831051 4386 catalog_manager.cc:1259] Loaded cluster ID: c7f486c5c0544604a757ab99b279479f
I20260806 02:52:46.831298 4386 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c: loading cluster ID for follower catalog manager: success
I20260806 02:52:46.835103 4386 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c: acquiring CA information for follower catalog manager: success
I20260806 02:52:46.837558 4386 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 02:52:47.106303 4387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:47.097373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:40381 (local address 127.0.0.1:43528)
0806 02:52:47.098847 (+ 1474us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:47.098868 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:47.099015 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:47.099527 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:47.099531 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:47.099557 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:52:47.099893 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:47.099903 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:47.101811 (+ 1908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:47.101815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:47.103841 (+ 2026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:47.103860 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:47.105601 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:47.105605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:47.105774 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:47.105801 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 02:52:47.105866 (+ 65us) client_negotiation.cc:240] Negotiation successful
0806 02:52:47.105973 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1329,"thread_start_us":1234,"threads_started":1}
I20260806 02:52:47.106406 4390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:47.097771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43528 (local address 127.25.211.188:40381)
0806 02:52:47.098684 (+ 913us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:47.098692 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:47.098977 (+ 285us) server_negotiation.cc:407] Connection header received
0806 02:52:47.099223 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:47.099229 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:47.099342 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:47.099499 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:52:47.100074 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:47.101639 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:47.104025 (+ 2386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:47.105370 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:47.105620 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:47.105828 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:47.105918 (+ 90us) server_negotiation.cc:299] Negotiation successful
0806 02:52:47.106064 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":737,"thread_start_us":564,"threads_started":1}
I20260806 02:52:47.107517 4333 raft_consensus.cc:1203] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260806 02:52:47.108126 4374 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5259274cdfee46ca8a88c8a0a0723c80. Latest consensus state: current_term: 1 leader_uuid: "5259274cdfee46ca8a88c8a0a0723c80" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5259274cdfee46ca8a88c8a0a0723c80" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45807 } } peers { permanent_uuid: "edad4ea4e4374f1383fb26cf1c316e03" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 44717 } attrs { promote: false } } peers { permanent_uuid: "f8a064e7081e4bc7b6da2b9c03c7e98c" member_type: NON_VOTER last_known_addr { host: "127.25.211.188" port: 40381 } attrs { promote: true } } }
I20260806 02:52:47.108428 4374 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c [sys.catalog]: This master's current role is: LEARNER
W20260806 02:52:47.186738 4366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:52:47.187135 4366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:52:47.187263 4366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:52:47.196421 4366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:52:47.196573 4366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:52:47.196753 4366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.129
I20260806 02:52:47.209340 4366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:40571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/krb5kdc/oryx_127.25.211.129.keytab
--principal=oryx/127.25.211.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.129:44003
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.211.129
--webserver_port=46839
--tserver_master_addrs=127.25.211.190:45807,127.25.211.189:44717,127.25.211.188:40381
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.129
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:52:47.211004 4366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:52:47.212533 4366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:52:47.229103 4406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:47.229007 4401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:47.229763 4399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:52:47.230764 4366 server_base.cc:1034] running on GCE node
I20260806 02:52:47.231477 4366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:52:47.232612 4366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:52:47.233841 4366 hybrid_clock.cc:648] HybridClock initialized: now 1785984767233777 us; error 64 us; skew 500 ppm
I20260806 02:52:47.234230 4366 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:52:47 dist-test-slave-zj8q krb5kdc[3054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984767, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.211.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:52:47.238392 4366 init.cc:377] Logged in from keytab as oryx/127.25.211.129@KRBTEST.COM (short username oryx)
I20260806 02:52:47.240249 4366 webserver.cc:460] Webserver started at http://127.25.211.129:46839/ using document root <none> and password file <none>
I20260806 02:52:47.241091 4366 fs_manager.cc:359] Metadata directory not provided
I20260806 02:52:47.241298 4366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:52:47.248054 4366 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:52:47.251300 4416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:47.252934 4366 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 02:52:47.253149 4366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-0/wal
uuid: "9c31aad209d04d6eb97f15899d7972dd"
format_stamp: "Formatted at 2026-08-06 02:52:43 on dist-test-slave-zj8q"
server_key: "d02e2e2b252fcb291035bcb5cf0cbde1"
server_key_iv: "a634f0cebc954f69c9ceb1d15e1d37cb"
server_key_version: "encryptionkey@0"
I20260806 02:52:47.254575 4366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:52:47.277585 4366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:52:47.282959 4366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:52:47.283465 4366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:52:47.284915 4366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:52:47.286779 4366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:52:47.286885 4366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:47.286996 4366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:52:47.287089 4366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:47.341423 4366 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.129:44003
I20260806 02:52:47.341537 4567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.129:44003 every 8 connection(s)
I20260806 02:52:47.343003 4366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
I20260806 02:52:47.350805 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 4366
I20260806 02:52:47.351235 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 3418
Aug 06 02:52:47 dist-test-slave-zj8q krb5kdc[3054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984767, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.211.129@KRBTEST.COM for oryx/127.25.211.189@KRBTEST.COM
Aug 06 02:52:47 dist-test-slave-zj8q krb5kdc[3054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984767, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.211.129@KRBTEST.COM for oryx/127.25.211.188@KRBTEST.COM
Aug 06 02:52:47 dist-test-slave-zj8q krb5kdc[3054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984767, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.211.129@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM
I20260806 02:52:47.382524 4390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:47.346323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:35811 (local address 127.25.211.188:40381)
0806 02:52:47.346531 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:47.346538 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:47.348442 (+ 1904us) server_negotiation.cc:407] Connection header received
0806 02:52:47.354771 (+ 6329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:47.354778 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:47.354885 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:47.354961 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:47.358244 (+ 3283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:47.359599 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:47.367053 (+ 7454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:47.367777 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:47.367897 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:47.372956 (+ 5059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:47.372965 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:47.372971 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:47.373017 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:47.377396 (+ 4379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:47.378535 (+ 1139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:47.378541 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:47.378544 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:47.378705 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:47.379657 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:47.379662 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:47.379666 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:47.380101 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:47.380296 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:47.381596 (+ 1300us) server_negotiation.cc:299] Negotiation successful
0806 02:52:47.381930 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260806 02:52:47.383769 4577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:47.347074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.189:44717 (local address 127.25.211.129:53169)
0806 02:52:47.349822 (+ 2748us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:47.349841 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:47.354029 (+ 4188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:47.360063 (+ 6034us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:47.360067 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:47.361483 (+ 1416us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:47.362268 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:47.362286 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:47.363725 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:47.363732 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:47.365246 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:47.365262 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:47.366429 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:47.366438 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:47.366781 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:47.367849 (+ 1068us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:52:47.367929 (+ 80us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:47.372552 (+ 4623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:47.378699 (+ 6147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:47.378703 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:47.378707 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:47.379573 (+ 866us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:47.380122 (+ 549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:47.380127 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:47.380130 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:47.380311 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:47.381019 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:47.381024 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:47.381246 (+ 222us) client_negotiation.cc:769] Sending connection context
0806 02:52:47.381437 (+ 191us) client_negotiation.cc:240] Negotiation successful
0806 02:52:47.382026 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2548,"mutex_wait_us":35,"spinlock_wait_cycles":40832,"thread_start_us":414,"threads_started":1}
I20260806 02:52:47.383903 4578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:47.347074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:40381 (local address 127.25.211.129:35811)
0806 02:52:47.348216 (+ 1142us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:47.348294 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:47.354594 (+ 6300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:47.355581 (+ 987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:47.355596 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:47.356660 (+ 1064us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:47.357898 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:47.357931 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:47.359860 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:47.359865 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:47.363365 (+ 3500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:47.363382 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:47.368361 (+ 4979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:47.368372 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:47.368662 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:47.369344 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:52:47.369372 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:47.372550 (+ 3178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:47.377600 (+ 5050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:47.377612 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:47.377632 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:47.378382 (+ 750us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:47.378934 (+ 552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:47.378939 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:47.378942 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:47.379511 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:47.380443 (+ 932us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:47.380452 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:47.381029 (+ 577us) client_negotiation.cc:769] Sending connection context
0806 02:52:47.382835 (+ 1806us) client_negotiation.cc:240] Negotiation successful
0806 02:52:47.383169 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":776,"thread_start_us":1896,"threads_started":1}
Aug 06 02:52:47 dist-test-slave-zj8q krb5kdc[3054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.25.211.130@KRBTEST.COM
I20260806 02:52:47.385344 4574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:47.346850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:53169 (local address 127.25.211.189:44717)
0806 02:52:47.359565 (+ 12715us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:47.359572 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:47.359600 (+ 28us) server_negotiation.cc:407] Connection header received
0806 02:52:47.359660 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:47.359665 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:47.359783 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:47.360384 (+ 601us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:47.362486 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:47.363547 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:47.365452 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:47.366086 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:47.367286 (+ 1200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:47.373033 (+ 5747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:47.373042 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:47.373047 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:47.373140 (+ 93us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:47.378510 (+ 5370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:47.379739 (+ 1229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:47.379745 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:47.379749 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:47.379971 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:47.380473 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:47.380478 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:47.380481 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:47.380858 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:47.384556 (+ 3698us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:47.384708 (+ 152us) server_negotiation.cc:299] Negotiation successful
0806 02:52:47.384966 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12575,"thread_start_us":13030,"threads_started":1}
I20260806 02:52:47.385586 4568 heartbeater.cc:347] Connected to a master server at 127.25.211.188:40381
I20260806 02:52:47.385923 4579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:47.349866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:45035 (local address 127.25.211.190:45807)
0806 02:52:47.350530 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:47.350536 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:47.351355 (+ 819us) server_negotiation.cc:407] Connection header received
0806 02:52:47.354665 (+ 3310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:47.354672 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:47.354774 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:47.354901 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:47.364508 (+ 9607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:47.365671 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:47.369939 (+ 4268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:47.370583 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:47.370688 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:47.376649 (+ 5961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:47.376658 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:47.376663 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:47.376709 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:47.381041 (+ 4332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:47.381951 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:47.381957 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:47.381961 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:47.382219 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:47.383059 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:47.383064 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:47.383067 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:47.383558 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:47.383687 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:47.384723 (+ 1036us) server_negotiation.cc:299] Negotiation successful
0806 02:52:47.384966 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":428,"threads_started":1}
I20260806 02:52:47.386083 4568 heartbeater.cc:464] Registering TS with master...
I20260806 02:52:47.387192 4568 heartbeater.cc:511] Master 127.25.211.188:40381 requested a full tablet report, sending...
I20260806 02:52:47.387161 4580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:47.350552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45807 (local address 127.25.211.129:45035)
0806 02:52:47.351244 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:47.351259 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:47.354509 (+ 3250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:47.358717 (+ 4208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:47.358723 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:47.363902 (+ 5179us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:47.364289 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:47.364301 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:47.365898 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:47.365903 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:47.369732 (+ 3829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:47.369750 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:47.371506 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:47.371517 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:47.372093 (+ 576us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:47.373017 (+ 924us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:52:47.373041 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:47.376384 (+ 3343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:47.381209 (+ 4825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:47.381213 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:47.381216 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:47.381808 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:47.382385 (+ 577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:47.382390 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:47.382393 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:47.382897 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:47.383785 (+ 888us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:47.383789 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:47.384423 (+ 634us) client_negotiation.cc:769] Sending connection context
0806 02:52:47.386241 (+ 1818us) client_negotiation.cc:240] Negotiation successful
0806 02:52:47.386545 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":408,"threads_started":1}
I20260806 02:52:47.388870 4569 heartbeater.cc:347] Connected to a master server at 127.25.211.190:45807
I20260806 02:52:47.389027 4569 heartbeater.cc:464] Registering TS with master...
I20260806 02:52:47.389051 4572 heartbeater.cc:347] Connected to a master server at 127.25.211.189:44717
I20260806 02:52:47.389250 4572 heartbeater.cc:464] Registering TS with master...
I20260806 02:52:47.389813 4569 heartbeater.cc:511] Master 127.25.211.190:45807 requested a full tablet report, sending...
I20260806 02:52:47.390107 4572 heartbeater.cc:511] Master 127.25.211.189:44717 requested a full tablet report, sending...
I20260806 02:52:47.390086 4312 ts_manager.cc:194] Registered new tserver with Master: 9c31aad209d04d6eb97f15899d7972dd (127.25.211.129:44003)
I20260806 02:52:47.391227 3120 ts_manager.cc:194] Re-registered known tserver with Master: 9c31aad209d04d6eb97f15899d7972dd (127.25.211.129:44003)
I20260806 02:52:47.392285 3120 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.25.211.129@KRBTEST.COM'} at 127.25.211.129:45035
I20260806 02:52:47.393930 4021 ts_manager.cc:194] Registered new tserver with Master: 9c31aad209d04d6eb97f15899d7972dd (127.25.211.129:44003)
I20260806 02:52:47.395632 4583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:47.352311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34348 (local address 127.25.211.130:42313)
0806 02:52:47.369546 (+ 17235us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:47.369556 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:47.369593 (+ 37us) server_negotiation.cc:407] Connection header received
0806 02:52:47.369697 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:47.369708 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:47.370014 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:47.370215 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:47.373299 (+ 3084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:47.375032 (+ 1733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:47.377680 (+ 2648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:47.378906 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:47.379038 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:47.386998 (+ 7960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:47.387021 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:47.387039 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:47.387098 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:47.391909 (+ 4811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:47.393094 (+ 1185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:47.393102 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:47.393104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:47.393299 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:47.393837 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:47.393841 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:47.393843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:47.394245 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:47.394469 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:47.394896 (+ 427us) server_negotiation.cc:299] Negotiation successful
0806 02:52:47.395106 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17035,"thread_start_us":18133,"threads_started":1}
I20260806 02:52:47.396903 3607 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:52:47.485795 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 3418
I20260806 02:52:47.513794 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.130:42313
--local_ip_for_outbound_sockets=127.25.211.130
--tserver_master_addrs=127.25.211.190:45807
--webserver_port=33273
--webserver_interface=127.25.211.130
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/krb5kdc/oryx_127.25.211.130.keytab
--principal=oryx/127.25.211.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:40571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.25.211.190:45807,127.25.211.189:44717
--tserver_master_addrs=127.25.211.190:45807,127.25.211.189:44717,127.25.211.188:40381 with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:52:47.628573 3956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5259274cdfee46ca8a88c8a0a0723c80 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5259274cdfee46ca8a88c8a0a0723c80" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5259274cdfee46ca8a88c8a0a0723c80" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45807 } } peers { permanent_uuid: "edad4ea4e4374f1383fb26cf1c316e03" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 44717 } attrs { promote: false } } peers { permanent_uuid: "f8a064e7081e4bc7b6da2b9c03c7e98c" member_type: NON_VOTER last_known_addr { host: "127.25.211.188" port: 40381 } attrs { promote: true } } }
I20260806 02:52:47.628973 3956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5259274cdfee46ca8a88c8a0a0723c80 [sys.catalog]: This master's current role is: LEADER
I20260806 02:52:47.630617 4589 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 5259274cdfee46ca8a88c8a0a0723c80: attempting to promote NON_VOTER f8a064e7081e4bc7b6da2b9c03c7e98c to VOTER
I20260806 02:52:47.631234 4589 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5259274cdfee46ca8a88c8a0a0723c80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5259274cdfee46ca8a88c8a0a0723c80" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45807 } } peers { permanent_uuid: "edad4ea4e4374f1383fb26cf1c316e03" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 44717 } attrs { promote: false } } peers { permanent_uuid: "f8a064e7081e4bc7b6da2b9c03c7e98c" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 40381 } attrs { promote: false } }
I20260806 02:52:47.633081 4033 raft_consensus.cc:1261] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03 [term 1 FOLLOWER]: Refusing update from remote peer 5259274cdfee46ca8a88c8a0a0723c80: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260806 02:52:47.633176 4333 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c [term 1 LEARNER]: Refusing update from remote peer 5259274cdfee46ca8a88c8a0a0723c80: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260806 02:52:47.633790 4589 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5259274cdfee46ca8a88c8a0a0723c80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edad4ea4e4374f1383fb26cf1c316e03" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 44717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260806 02:52:47.634261 4590 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5259274cdfee46ca8a88c8a0a0723c80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8a064e7081e4bc7b6da2b9c03c7e98c" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 40381 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260806 02:52:47.638504 3956 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5259274cdfee46ca8a88c8a0a0723c80 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f8a064e7081e4bc7b6da2b9c03c7e98c (127.25.211.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5259274cdfee46ca8a88c8a0a0723c80" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45807 } } peers { permanent_uuid: "edad4ea4e4374f1383fb26cf1c316e03" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 44717 } attrs { promote: false } } peers { permanent_uuid: "f8a064e7081e4bc7b6da2b9c03c7e98c" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 40381 } attrs { promote: false } } }
I20260806 02:52:47.641506 4589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5259274cdfee46ca8a88c8a0a0723c80 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5259274cdfee46ca8a88c8a0a0723c80" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5259274cdfee46ca8a88c8a0a0723c80" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45807 } } peers { permanent_uuid: "edad4ea4e4374f1383fb26cf1c316e03" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 44717 } attrs { promote: false } } peers { permanent_uuid: "f8a064e7081e4bc7b6da2b9c03c7e98c" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 40381 } attrs { promote: false } } }
I20260806 02:52:47.641916 4589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5259274cdfee46ca8a88c8a0a0723c80 [sys.catalog]: This master's current role is: LEADER
I20260806 02:52:47.642366 4033 raft_consensus.cc:2932] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f8a064e7081e4bc7b6da2b9c03c7e98c (127.25.211.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5259274cdfee46ca8a88c8a0a0723c80" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45807 } } peers { permanent_uuid: "edad4ea4e4374f1383fb26cf1c316e03" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 44717 } attrs { promote: false } } peers { permanent_uuid: "f8a064e7081e4bc7b6da2b9c03c7e98c" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 40381 } attrs { promote: false } } }
I20260806 02:52:47.642781 4333 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f8a064e7081e4bc7b6da2b9c03c7e98c (127.25.211.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5259274cdfee46ca8a88c8a0a0723c80" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45807 } } peers { permanent_uuid: "edad4ea4e4374f1383fb26cf1c316e03" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 44717 } attrs { promote: false } } peers { permanent_uuid: "f8a064e7081e4bc7b6da2b9c03c7e98c" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 40381 } attrs { promote: false } } }
I20260806 02:52:47.645433 3956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5259274cdfee46ca8a88c8a0a0723c80 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5259274cdfee46ca8a88c8a0a0723c80" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5259274cdfee46ca8a88c8a0a0723c80" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45807 } } peers { permanent_uuid: "edad4ea4e4374f1383fb26cf1c316e03" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 44717 } attrs { promote: false } } peers { permanent_uuid: "f8a064e7081e4bc7b6da2b9c03c7e98c" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 40381 } attrs { promote: false } } }
I20260806 02:52:47.645772 3956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5259274cdfee46ca8a88c8a0a0723c80 [sys.catalog]: This master's current role is: LEADER
I20260806 02:52:47.646057 4590 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5259274cdfee46ca8a88c8a0a0723c80 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5259274cdfee46ca8a88c8a0a0723c80" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5259274cdfee46ca8a88c8a0a0723c80" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45807 } } peers { permanent_uuid: "edad4ea4e4374f1383fb26cf1c316e03" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 44717 } attrs { promote: false } } peers { permanent_uuid: "f8a064e7081e4bc7b6da2b9c03c7e98c" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 40381 } attrs { promote: false } } }
I20260806 02:52:47.646368 4590 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5259274cdfee46ca8a88c8a0a0723c80 [sys.catalog]: This master's current role is: LEADER
I20260806 02:52:47.655140 4597 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5259274cdfee46ca8a88c8a0a0723c80" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5259274cdfee46ca8a88c8a0a0723c80" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45807 } } peers { permanent_uuid: "edad4ea4e4374f1383fb26cf1c316e03" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 44717 } attrs { promote: false } } peers { permanent_uuid: "f8a064e7081e4bc7b6da2b9c03c7e98c" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 40381 } attrs { promote: false } } }
I20260806 02:52:47.655959 4597 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:52:47.656662 4596 sys_catalog.cc:455] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5259274cdfee46ca8a88c8a0a0723c80" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5259274cdfee46ca8a88c8a0a0723c80" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45807 } } peers { permanent_uuid: "edad4ea4e4374f1383fb26cf1c316e03" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 44717 } attrs { promote: false } } peers { permanent_uuid: "f8a064e7081e4bc7b6da2b9c03c7e98c" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 40381 } attrs { promote: false } } }
I20260806 02:52:47.657140 4596 sys_catalog.cc:458] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03 [sys.catalog]: This master's current role is: FOLLOWER
W20260806 02:52:47.874521 4588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:52:47.874913 4588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:52:47.875072 4588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:52:47.883240 4588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:52:47.883387 4588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:52:47.883544 4588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.130
I20260806 02:52:47.893997 4588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:40571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/krb5kdc/oryx_127.25.211.130.keytab
--principal=oryx/127.25.211.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.130:42313
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.211.130
--webserver_port=33273
--tserver_master_addrs=127.25.211.190:45807,127.25.211.189:44717,127.25.211.188:40381
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.130
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:52:47.895515 4588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:52:47.896898 4588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:52:47.910931 4607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:47.910955 4605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:47.911479 4611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:52:47.911966 4588 server_base.cc:1034] running on GCE node
I20260806 02:52:47.912784 4588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:52:47.914026 4588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:52:47.915277 4588 hybrid_clock.cc:648] HybridClock initialized: now 1785984767915204 us; error 61 us; skew 500 ppm
I20260806 02:52:47.915679 4588 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:52:47 dist-test-slave-zj8q krb5kdc[3054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984767, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.211.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:52:47.919935 4588 init.cc:377] Logged in from keytab as oryx/127.25.211.130@KRBTEST.COM (short username oryx)
I20260806 02:52:47.922025 4588 webserver.cc:460] Webserver started at http://127.25.211.130:33273/ using document root <none> and password file <none>
I20260806 02:52:47.923002 4588 fs_manager.cc:359] Metadata directory not provided
I20260806 02:52:47.923167 4588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:52:47.929636 4588 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260806 02:52:47.932765 4622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:47.934270 4588 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:52:47.934422 4588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-1/wal
uuid: "916e85b805ce4cbab170fdf74b250f9e"
format_stamp: "Formatted at 2026-08-06 02:52:44 on dist-test-slave-zj8q"
server_key: "f29434c91229076829751b305685c169"
server_key_iv: "cbc9b745c10384d9ccda2ddf6d7ce657"
server_key_version: "encryptionkey@0"
I20260806 02:52:47.935474 4588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:52:47.960667 4588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:52:47.965178 4588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:52:47.965461 4588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:52:47.966702 4588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:52:47.968227 4588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:52:47.968324 4588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:47.968410 4588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:52:47.968463 4588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:48.000917 4588 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.130:42313
I20260806 02:52:48.000976 4779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.130:42313 every 8 connection(s)
I20260806 02:52:48.002573 4588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
I20260806 02:52:48.003875 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 4588
I20260806 02:52:48.004184 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 3625
Aug 06 02:52:48 dist-test-slave-zj8q krb5kdc[3054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984767, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.211.130@KRBTEST.COM for oryx/127.25.211.188@KRBTEST.COM
Aug 06 02:52:48 dist-test-slave-zj8q krb5kdc[3054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984767, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.211.130@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM
Aug 06 02:52:48 dist-test-slave-zj8q krb5kdc[3054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984767, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.211.130@KRBTEST.COM for oryx/127.25.211.189@KRBTEST.COM
Aug 06 02:52:48 dist-test-slave-zj8q krb5kdc[3054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.25.211.131@KRBTEST.COM
I20260806 02:52:48.069895 4798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:48.024978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:33009 (local address 127.25.211.188:40381)
0806 02:52:48.040494 (+ 15516us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:48.040503 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:48.043058 (+ 2555us) server_negotiation.cc:407] Connection header received
0806 02:52:48.043298 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:48.043304 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:48.043401 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:48.043513 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:48.047835 (+ 4322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:48.049687 (+ 1852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:48.052128 (+ 2441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:48.052869 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:48.052989 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:48.059465 (+ 6476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:48.059471 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:48.059475 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:48.059514 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:48.064516 (+ 5002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:48.066672 (+ 2156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:48.066678 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:48.066681 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:48.067205 (+ 524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:48.067612 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:48.067618 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:48.067621 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:48.067996 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:48.068149 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:48.069196 (+ 1047us) server_negotiation.cc:299] Negotiation successful
0806 02:52:48.069448 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15331,"thread_start_us":15526,"threads_started":1}
I20260806 02:52:48.070969 4788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:48.005639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:55333 (local address 127.25.211.190:45807)
0806 02:52:48.044156 (+ 38517us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:48.044161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:48.044180 (+ 19us) server_negotiation.cc:407] Connection header received
0806 02:52:48.044238 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:48.044242 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:48.044335 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:48.044450 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:48.046950 (+ 2500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:48.048199 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:48.051646 (+ 3447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:48.052457 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:48.052593 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:48.059187 (+ 6594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:48.059195 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:48.059199 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:48.059256 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:48.063278 (+ 4022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:48.066913 (+ 3635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:48.066918 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:48.066922 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:48.067105 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:48.067994 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:48.067999 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:48.068001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:48.068371 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:48.068500 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:48.070335 (+ 1835us) server_negotiation.cc:299] Negotiation successful
0806 02:52:48.070591 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38379,"thread_start_us":38234,"threads_started":1}
I20260806 02:52:48.071214 4797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:48.007373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.189:44717 (local address 127.25.211.130:52867)
0806 02:52:48.041452 (+ 34079us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:48.041478 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:48.042000 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:48.045431 (+ 3431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:48.045440 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:48.048067 (+ 2627us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:48.048592 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:48.048607 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:48.050097 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:48.050102 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:48.052048 (+ 1946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:48.052064 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:48.054205 (+ 2141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:48.054214 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:48.054481 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:48.055629 (+ 1148us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:52:48.055655 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:48.059943 (+ 4288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:48.064766 (+ 4823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:48.064770 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:48.064774 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:48.066499 (+ 1725us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:48.067039 (+ 540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:48.067044 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:48.067047 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:48.067394 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:48.068318 (+ 924us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:48.068322 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:48.068762 (+ 440us) client_negotiation.cc:769] Sending connection context
0806 02:52:48.069043 (+ 281us) client_negotiation.cc:240] Negotiation successful
0806 02:52:48.069676 (+ 633us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33876,"mutex_wait_us":74,"thread_start_us":34236,"threads_started":1}
I20260806 02:52:48.071839 4794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:48.024795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:40381 (local address 127.25.211.130:33009)
0806 02:52:48.042928 (+ 18133us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:48.042951 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:48.043157 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:48.044236 (+ 1079us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:48.044246 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:48.046264 (+ 2018us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:48.047417 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:48.047433 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:48.049862 (+ 2429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:48.049867 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:48.051925 (+ 2058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:48.051940 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:48.053278 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:48.053285 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:48.053430 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:48.054479 (+ 1049us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:52:48.054530 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:48.059210 (+ 4680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:48.064708 (+ 5498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:48.064719 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:48.064743 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:48.066499 (+ 1756us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:48.067322 (+ 823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:48.067327 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:48.067331 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:48.067476 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:48.068234 (+ 758us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:48.068244 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:48.068768 (+ 524us) client_negotiation.cc:769] Sending connection context
0806 02:52:48.069044 (+ 276us) client_negotiation.cc:240] Negotiation successful
0806 02:52:48.069927 (+ 883us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17949,"mutex_wait_us":425,"spinlock_wait_cycles":34816,"thread_start_us":16224,"threads_started":1}
I20260806 02:52:48.072275 4792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:48.007004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:52867 (local address 127.25.211.189:44717)
0806 02:52:48.042113 (+ 35109us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:48.042121 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:48.043629 (+ 1508us) server_negotiation.cc:407] Connection header received
0806 02:52:48.043707 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:48.043711 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:48.043802 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:48.043918 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:48.048785 (+ 4867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:48.049899 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:48.052238 (+ 2339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:48.052981 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:48.053083 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:48.060258 (+ 7175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:48.060268 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:48.060273 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:48.060329 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:48.064644 (+ 4315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:48.066662 (+ 2018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:48.066668 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:48.066671 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:48.066892 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:48.067523 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:48.067528 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:48.067532 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:48.068016 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:48.068249 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:48.071363 (+ 3114us) server_negotiation.cc:299] Negotiation successful
0806 02:52:48.072010 (+ 647us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34959,"thread_start_us":34816,"threads_started":1}
I20260806 02:52:48.072633 4791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:48.006347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45807 (local address 127.25.211.130:55333)
0806 02:52:48.021260 (+ 14913us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:48.021338 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:48.028254 (+ 6916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:48.044517 (+ 16263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:48.044521 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:48.045434 (+ 913us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:48.046751 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:48.046784 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:48.048381 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:48.048387 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:48.050812 (+ 2425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:48.050828 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:48.052973 (+ 2145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:48.052981 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:48.053320 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:48.054562 (+ 1242us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:52:48.054600 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:48.058870 (+ 4270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:48.064794 (+ 5924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:48.064798 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:48.064801 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:48.065796 (+ 995us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:48.067491 (+ 1695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:48.067494 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:48.067496 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:48.067647 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:48.069236 (+ 1589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:48.069240 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:48.069429 (+ 189us) client_negotiation.cc:769] Sending connection context
0806 02:52:48.069547 (+ 118us) client_negotiation.cc:240] Negotiation successful
0806 02:52:48.069791 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14424,"mutex_wait_us":2,"spinlock_wait_cycles":39552,"thread_start_us":16923,"threads_started":1}
I20260806 02:52:48.072813 4790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:48.006506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57556 (local address 127.25.211.131:39997)
0806 02:52:48.044425 (+ 37919us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:48.044434 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:48.044466 (+ 32us) server_negotiation.cc:407] Connection header received
0806 02:52:48.044537 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:48.044545 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:48.044899 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:48.045065 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:48.047414 (+ 2349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:48.049326 (+ 1912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:48.051493 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:48.052365 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:48.053359 (+ 994us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:48.061286 (+ 7927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:48.061301 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:48.061316 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:48.061352 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:48.066350 (+ 4998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:48.067649 (+ 1299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:48.067660 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:48.067663 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:48.067829 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:48.068324 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:48.068329 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:48.068332 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:48.068821 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:48.069153 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:48.071968 (+ 2815us) server_negotiation.cc:299] Negotiation successful
0806 02:52:48.072291 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37697,"thread_start_us":36955,"threads_started":1}
I20260806 02:52:48.073439 4783 heartbeater.cc:347] Connected to a master server at 127.25.211.189:44717
I20260806 02:52:48.073588 4780 heartbeater.cc:347] Connected to a master server at 127.25.211.188:40381
I20260806 02:52:48.073856 4783 heartbeater.cc:464] Registering TS with master...
I20260806 02:52:48.073863 4780 heartbeater.cc:464] Registering TS with master...
I20260806 02:52:48.074892 4780 heartbeater.cc:511] Master 127.25.211.188:40381 requested a full tablet report, sending...
I20260806 02:52:48.074918 4783 heartbeater.cc:511] Master 127.25.211.189:44717 requested a full tablet report, sending...
I20260806 02:52:48.074939 3790 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:52:48.075431 4782 heartbeater.cc:347] Connected to a master server at 127.25.211.190:45807
I20260806 02:52:48.075768 4782 heartbeater.cc:464] Registering TS with master...
I20260806 02:52:48.076503 4782 heartbeater.cc:511] Master 127.25.211.190:45807 requested a full tablet report, sending...
I20260806 02:52:48.077061 4021 ts_manager.cc:194] Registered new tserver with Master: 916e85b805ce4cbab170fdf74b250f9e (127.25.211.130:42313)
I20260806 02:52:48.077158 4312 ts_manager.cc:194] Registered new tserver with Master: 916e85b805ce4cbab170fdf74b250f9e (127.25.211.130:42313)
I20260806 02:52:48.077760 3120 ts_manager.cc:194] Re-registered known tserver with Master: 916e85b805ce4cbab170fdf74b250f9e (127.25.211.130:42313)
I20260806 02:52:48.078814 3120 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.25.211.130@KRBTEST.COM'} at 127.25.211.130:55333
I20260806 02:52:48.167595 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 3625
I20260806 02:52:48.192490 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.131:39997
--local_ip_for_outbound_sockets=127.25.211.131
--tserver_master_addrs=127.25.211.190:45807
--webserver_port=33695
--webserver_interface=127.25.211.131
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/krb5kdc/oryx_127.25.211.131.keytab
--principal=oryx/127.25.211.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:40571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.25.211.190:45807,127.25.211.189:44717
--tserver_master_addrs=127.25.211.190:45807,127.25.211.189:44717,127.25.211.188:40381 with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:52:48.396482 4569 heartbeater.cc:503] Master 127.25.211.190:45807 was elected leader, sending a full tablet report...
W20260806 02:52:48.550698 4801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:52:48.550998 4801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:52:48.551118 4801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:52:48.558527 4801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:52:48.558647 4801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:52:48.558800 4801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.131
I20260806 02:52:48.569116 4801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:40571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/krb5kdc/oryx_127.25.211.131.keytab
--principal=oryx/127.25.211.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.131:39997
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.211.131
--webserver_port=33695
--tserver_master_addrs=127.25.211.190:45807,127.25.211.189:44717,127.25.211.188:40381
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.131
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:52:48.570572 4801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:52:48.572021 4801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:52:48.586234 4812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:48.586346 4814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:48.586367 4810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:52:48.587399 4801 server_base.cc:1034] running on GCE node
I20260806 02:52:48.588084 4801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:52:48.589105 4801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:52:48.590306 4801 hybrid_clock.cc:648] HybridClock initialized: now 1785984768590255 us; error 48 us; skew 500 ppm
I20260806 02:52:48.590627 4801 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:52:48 dist-test-slave-zj8q krb5kdc[3054](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984768, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.211.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:52:48.593871 4801 init.cc:377] Logged in from keytab as oryx/127.25.211.131@KRBTEST.COM (short username oryx)
I20260806 02:52:48.595512 4801 webserver.cc:460] Webserver started at http://127.25.211.131:33695/ using document root <none> and password file <none>
I20260806 02:52:48.596216 4801 fs_manager.cc:359] Metadata directory not provided
I20260806 02:52:48.596347 4801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:52:48.602123 4801 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.003s
I20260806 02:52:48.605139 4824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:48.606725 4801 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260806 02:52:48.606905 4801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-2/wal
uuid: "72d6e066b33b4d3089f69ac997a038e2"
format_stamp: "Formatted at 2026-08-06 02:52:44 on dist-test-slave-zj8q"
server_key: "a22224be660e706d6abfd65c5ceeb42c"
server_key_iv: "e6171d925ada3fd50936e7b7dc151c0e"
server_key_version: "encryptionkey@0"
I20260806 02:52:48.607954 4801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:52:48.621382 4801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:52:48.626024 4801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:52:48.626331 4801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:52:48.627326 4801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:52:48.628702 4801 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:52:48.628793 4801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:48.628875 4801 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:52:48.628931 4801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:48.660820 4801 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.131:39997
I20260806 02:52:48.660881 4989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.131:39997 every 8 connection(s)
I20260806 02:52:48.662452 4801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
I20260806 02:52:48.669296 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 4801
Aug 06 02:52:48 dist-test-slave-zj8q krb5kdc[3054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984768, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.211.131@KRBTEST.COM for oryx/127.25.211.188@KRBTEST.COM
Aug 06 02:52:48 dist-test-slave-zj8q krb5kdc[3054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984768, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.211.131@KRBTEST.COM for oryx/127.25.211.189@KRBTEST.COM
I20260806 02:52:48.694687 5006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:48.667393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:40879 (local address 127.25.211.189:44717)
0806 02:52:48.667874 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:48.667881 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:48.672320 (+ 4439us) server_negotiation.cc:407] Connection header received
0806 02:52:48.672399 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:48.672405 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:48.672507 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:48.672601 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:48.674131 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:48.675348 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:48.678783 (+ 3435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:48.679662 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:48.679797 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:48.685699 (+ 5902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:48.685707 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:48.685713 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:48.685764 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:48.689774 (+ 4010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:48.691036 (+ 1262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:48.691043 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:48.691048 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:48.691271 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:48.691861 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:48.691867 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:48.691871 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:48.692583 (+ 712us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:48.692751 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:48.693934 (+ 1183us) server_negotiation.cc:299] Negotiation successful
0806 02:52:48.694218 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":246,"threads_started":1}
I20260806 02:52:48.696426 5004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:48.666548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:40381 (local address 127.25.211.131:57303)
0806 02:52:48.667589 (+ 1041us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:48.667654 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:48.670473 (+ 2819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:48.671818 (+ 1345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:48.671838 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:48.672812 (+ 974us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:48.673905 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:48.673922 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:48.675725 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:48.675732 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:48.679312 (+ 3580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:48.679326 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:48.680694 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:48.680701 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:48.680914 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:48.681531 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:52:48.681562 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:48.685324 (+ 3762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:48.691962 (+ 6638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:48.691967 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:48.691971 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:48.692462 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:48.692957 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:48.692961 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:48.692964 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:48.693282 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:48.694002 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:48.694007 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:48.694213 (+ 206us) client_negotiation.cc:769] Sending connection context
0806 02:52:48.694344 (+ 131us) client_negotiation.cc:240] Negotiation successful
0806 02:52:48.694634 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"mutex_wait_us":888,"spinlock_wait_cycles":2304,"thread_start_us":876,"threads_started":1}
I20260806 02:52:48.696625 5007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:48.667395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.189:44717 (local address 127.25.211.131:40879)
0806 02:52:48.672016 (+ 4621us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:48.672036 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:48.672177 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:48.672785 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:48.672790 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:48.673451 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:48.673888 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:48.673917 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:48.675871 (+ 1954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:48.675877 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:48.678550 (+ 2673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:48.678574 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:48.680182 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:48.680190 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:48.680560 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:48.681419 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:52:48.681468 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:48.685316 (+ 3848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:48.689978 (+ 4662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:48.689992 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:48.690022 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:48.690858 (+ 836us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:48.691413 (+ 555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:48.691419 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:48.691423 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:48.691718 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:48.692838 (+ 1120us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:48.692849 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:48.693434 (+ 585us) client_negotiation.cc:769] Sending connection context
0806 02:52:48.693783 (+ 349us) client_negotiation.cc:240] Negotiation successful
0806 02:52:48.694412 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4487,"mutex_wait_us":65,"spinlock_wait_cycles":7552,"thread_start_us":4329,"threads_started":1}
I20260806 02:52:48.698325 5002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:48.665796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:57303 (local address 127.25.211.188:40381)
0806 02:52:48.666325 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:48.666330 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:48.669579 (+ 3249us) server_negotiation.cc:407] Connection header received
0806 02:52:48.671030 (+ 1451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:48.671038 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:48.671152 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:48.671263 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:48.674143 (+ 2880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:48.675457 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:48.679494 (+ 4037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:48.680209 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:48.680318 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:48.685781 (+ 5463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:48.685790 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:48.685795 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:48.685846 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:48.689707 (+ 3861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:48.692637 (+ 2930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:48.692643 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:48.692646 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:48.692804 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:48.693438 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:48.693442 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:48.693445 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:48.693799 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:48.693939 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:48.697377 (+ 3438us) server_negotiation.cc:299] Negotiation successful
0806 02:52:48.697685 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":266,"threads_started":1}
I20260806 02:52:48.698521 4998 heartbeater.cc:347] Connected to a master server at 127.25.211.189:44717
I20260806 02:52:48.699043 4998 heartbeater.cc:464] Registering TS with master...
I20260806 02:52:48.699455 4991 heartbeater.cc:347] Connected to a master server at 127.25.211.188:40381
I20260806 02:52:48.699630 4991 heartbeater.cc:464] Registering TS with master...
I20260806 02:52:48.700453 4991 heartbeater.cc:511] Master 127.25.211.188:40381 requested a full tablet report, sending...
Aug 06 02:52:48 dist-test-slave-zj8q krb5kdc[3054](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984768, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.211.131@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM
I20260806 02:52:48.701638 4998 heartbeater.cc:511] Master 127.25.211.189:44717 requested a full tablet report, sending...
I20260806 02:52:48.703084 4021 ts_manager.cc:194] Registered new tserver with Master: 72d6e066b33b4d3089f69ac997a038e2 (127.25.211.131:39997)
I20260806 02:52:48.703207 4312 ts_manager.cc:194] Registered new tserver with Master: 72d6e066b33b4d3089f69ac997a038e2 (127.25.211.131:39997)
I20260806 02:52:48.710776 5010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:48.672011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45807 (local address 127.25.211.131:53171)
0806 02:52:48.678863 (+ 6852us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:48.678888 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:48.679033 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:48.685582 (+ 6549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:48.685589 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:48.686271 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:48.686662 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:48.686678 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:48.688571 (+ 1893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:48.688578 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:48.690380 (+ 1802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:48.690396 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:48.691734 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:48.691742 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:48.692004 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:48.692633 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:52:48.692655 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:48.702433 (+ 9778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:48.706993 (+ 4560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:48.707000 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:48.707004 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:48.707686 (+ 682us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:48.708353 (+ 667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:48.708359 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:48.708364 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:48.708686 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:48.709466 (+ 780us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:48.709472 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:48.709657 (+ 185us) client_negotiation.cc:769] Sending connection context
0806 02:52:48.709788 (+ 131us) client_negotiation.cc:240] Negotiation successful
0806 02:52:48.710177 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6696,"thread_start_us":6513,"threads_started":1}
I20260806 02:52:48.711843 5008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:48.668850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:53171 (local address 127.25.211.190:45807)
0806 02:52:48.684683 (+ 15833us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:48.684690 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:48.684712 (+ 22us) server_negotiation.cc:407] Connection header received
0806 02:52:48.684785 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:48.684791 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:48.684881 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:48.686848 (+ 1967us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:48.687083 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:48.688202 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:48.690578 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:48.691454 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:48.692331 (+ 877us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:48.702706 (+ 10375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:48.702714 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:48.702718 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:48.702766 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:48.706824 (+ 4058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:48.707890 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:48.707896 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:48.707900 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:48.708203 (+ 303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:48.708852 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:48.708857 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:48.708861 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:48.709315 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:48.711045 (+ 1730us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:48.711227 (+ 182us) server_negotiation.cc:299] Negotiation successful
0806 02:52:48.711476 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15684,"thread_start_us":16288,"threads_started":1}
I20260806 02:52:48.712986 4996 heartbeater.cc:347] Connected to a master server at 127.25.211.190:45807
I20260806 02:52:48.713164 4996 heartbeater.cc:464] Registering TS with master...
I20260806 02:52:48.714044 4996 heartbeater.cc:511] Master 127.25.211.190:45807 requested a full tablet report, sending...
I20260806 02:52:48.715286 3120 ts_manager.cc:194] Re-registered known tserver with Master: 72d6e066b33b4d3089f69ac997a038e2 (127.25.211.131:39997)
I20260806 02:52:48.716391 3120 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.25.211.131@KRBTEST.COM'} at 127.25.211.131:53171
I20260806 02:52:48.725096 5006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:48.695613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54112 (local address 127.25.211.189:44717)
0806 02:52:48.695803 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:48.695810 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:48.701785 (+ 5975us) server_negotiation.cc:407] Connection header received
0806 02:52:48.702247 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:48.702252 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:48.702338 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:48.702433 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:48.705249 (+ 2816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:48.706046 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:48.708121 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:48.709093 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:48.709198 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:48.717579 (+ 8381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:48.717587 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:48.717591 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:48.717638 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:48.721707 (+ 4069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:48.722765 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:48.722770 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:48.722775 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:48.722938 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:48.723472 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:48.723478 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:48.723482 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:48.723835 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:48.724142 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:48.724630 (+ 488us) server_negotiation.cc:299] Negotiation successful
0806 02:52:48.724863 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 02:52:48.728664 5018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:48.697298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41216 (local address 127.25.211.190:45807)
0806 02:52:48.710616 (+ 13318us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:48.710623 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:48.710642 (+ 19us) server_negotiation.cc:407] Connection header received
0806 02:52:48.710690 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:48.710694 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:48.710765 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:48.710881 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:48.713918 (+ 3037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:48.714911 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:48.716487 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:48.717226 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:48.717338 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:48.721007 (+ 3669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:48.721015 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:48.721020 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:48.721115 (+ 95us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:48.725054 (+ 3939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:48.726133 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:48.726139 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:48.726142 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:48.726365 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:48.726886 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:48.726892 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:48.726895 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:48.727335 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:48.727624 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:48.727974 (+ 350us) server_negotiation.cc:299] Negotiation successful
0806 02:52:48.728204 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13189,"thread_start_us":13005,"threads_started":1}
I20260806 02:52:48.729792 5002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:48.705117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43544 (local address 127.25.211.188:40381)
0806 02:52:48.705329 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:48.705334 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:48.707578 (+ 2244us) server_negotiation.cc:407] Connection header received
0806 02:52:48.707834 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:48.707841 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:48.707977 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:48.708049 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:48.714289 (+ 6240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:48.715197 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:48.718535 (+ 3338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:48.719277 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:48.719384 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:48.722517 (+ 3133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:48.722522 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:48.722525 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:48.722560 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:48.726737 (+ 4177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:48.727423 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:48.727429 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:48.727433 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:48.727635 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:48.728062 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:48.728067 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:48.728070 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:48.728498 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:48.728825 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:48.729178 (+ 353us) server_negotiation.cc:299] Negotiation successful
0806 02:52:48.729455 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 02:52:48.732861 3120 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41216:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:52:48.741941 3120 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:52:48.784803 5047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:48.767033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.130:42313 (local address 127.0.0.1:34360)
0806 02:52:48.767849 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:48.767866 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:48.767976 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:48.776256 (+ 8280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:48.776263 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:48.776300 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:52:48.776979 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:48.776995 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:48.779207 (+ 2212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:48.779212 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:48.781684 (+ 2472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:48.781698 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:48.783679 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:48.783686 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:48.783910 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:48.783947 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 02:52:48.784040 (+ 93us) client_negotiation.cc:240] Negotiation successful
0806 02:52:48.784188 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"thread_start_us":9731,"threads_started":1}
I20260806 02:52:48.785632 5048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:48.767565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34360 (local address 127.25.211.130:42313)
0806 02:52:48.775339 (+ 7774us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:48.775351 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:48.775389 (+ 38us) server_negotiation.cc:407] Connection header received
0806 02:52:48.775569 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:48.775579 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:48.775902 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:48.776234 (+ 332us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:52:48.777216 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:48.779014 (+ 1798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:48.781861 (+ 2847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:48.783395 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:48.783571 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:48.783770 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:48.785158 (+ 1388us) server_negotiation.cc:299] Negotiation successful
0806 02:52:48.785356 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7545,"thread_start_us":7422,"threads_started":1}
I20260806 02:52:48.789773 5043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:48.766187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:44003 (local address 127.0.0.1:39464)
0806 02:52:48.767128 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:48.767155 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:48.767284 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:48.779988 (+ 12704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:48.779994 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:48.780021 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:52:48.782240 (+ 2219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:48.782252 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:48.785089 (+ 2837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:48.785094 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:48.786991 (+ 1897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:48.787008 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:48.788897 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:48.788904 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:48.789087 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:48.789123 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 02:52:48.789198 (+ 75us) client_negotiation.cc:240] Negotiation successful
0806 02:52:48.789348 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774,"thread_start_us":733,"threads_started":1}
I20260806 02:52:48.790956 5044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:48.766422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39464 (local address 127.25.211.129:44003)
0806 02:52:48.779297 (+ 12875us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:48.779305 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:48.779336 (+ 31us) server_negotiation.cc:407] Connection header received
0806 02:52:48.779440 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:48.779447 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:48.779743 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:48.780174 (+ 431us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:52:48.782482 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:48.784916 (+ 2434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:48.787213 (+ 2297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:48.788655 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:48.790274 (+ 1619us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:48.790420 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:48.790482 (+ 62us) server_negotiation.cc:299] Negotiation successful
0806 02:52:48.790652 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12698,"thread_start_us":12580,"threads_started":1}
I20260806 02:52:48.791107 4692 tablet_service.cc:1467] Processing CreateTablet for tablet 95365a93b5554c37b6e7364b9ab4d03c (DEFAULT_TABLE table=test-table [id=9ddafece9326495daa29c991d3bb2a59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:52:48.795189 4692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95365a93b5554c37b6e7364b9ab4d03c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:48.795120 4484 tablet_service.cc:1467] Processing CreateTablet for tablet 95365a93b5554c37b6e7364b9ab4d03c (DEFAULT_TABLE table=test-table [id=9ddafece9326495daa29c991d3bb2a59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:52:48.796744 4484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95365a93b5554c37b6e7364b9ab4d03c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:48.803558 5046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:48.767261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.131:39997 (local address 127.0.0.1:57572)
0806 02:52:48.777193 (+ 9932us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:48.777214 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:48.777451 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:48.778573 (+ 1122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:48.778577 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:48.778604 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:52:48.779278 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:48.779292 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:48.781631 (+ 2339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:48.781637 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:48.799316 (+ 17679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:48.799338 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:48.802518 (+ 3180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:48.802526 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:48.802751 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:48.802789 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 02:52:48.802901 (+ 112us) client_negotiation.cc:240] Negotiation successful
0806 02:52:48.803080 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9765,"thread_start_us":1245,"threads_started":1}
I20260806 02:52:48.803596 5050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:48.768150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57572 (local address 127.25.211.131:39997)
0806 02:52:48.769304 (+ 1154us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:48.769314 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:48.777337 (+ 8023us) server_negotiation.cc:407] Connection header received
0806 02:52:48.777641 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:48.777649 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:48.777972 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:48.778205 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:52:48.779564 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:48.781494 (+ 1930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:48.799574 (+ 18080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:48.801793 (+ 2219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:48.801975 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:48.802208 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:48.803016 (+ 808us) server_negotiation.cc:299] Negotiation successful
0806 02:52:48.803229 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":883,"thread_start_us":581,"threads_started":1}
I20260806 02:52:48.808573 4892 tablet_service.cc:1467] Processing CreateTablet for tablet 95365a93b5554c37b6e7364b9ab4d03c (DEFAULT_TABLE table=test-table [id=9ddafece9326495daa29c991d3bb2a59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:52:48.810341 4892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95365a93b5554c37b6e7364b9ab4d03c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:48.813130 5052 tablet_bootstrap.cc:492] T 95365a93b5554c37b6e7364b9ab4d03c P 9c31aad209d04d6eb97f15899d7972dd: Bootstrap starting.
I20260806 02:52:48.817332 5052 tablet_bootstrap.cc:654] T 95365a93b5554c37b6e7364b9ab4d03c P 9c31aad209d04d6eb97f15899d7972dd: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:48.818871 5053 tablet_bootstrap.cc:492] T 95365a93b5554c37b6e7364b9ab4d03c P 916e85b805ce4cbab170fdf74b250f9e: Bootstrap starting.
I20260806 02:52:48.820070 5052 log.cc:824] T 95365a93b5554c37b6e7364b9ab4d03c P 9c31aad209d04d6eb97f15899d7972dd: Log is configured to *not* fsync() on all Append() calls
I20260806 02:52:48.823596 5053 tablet_bootstrap.cc:654] T 95365a93b5554c37b6e7364b9ab4d03c P 916e85b805ce4cbab170fdf74b250f9e: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:48.824242 5052 tablet_bootstrap.cc:492] T 95365a93b5554c37b6e7364b9ab4d03c P 9c31aad209d04d6eb97f15899d7972dd: No bootstrap required, opened a new log
I20260806 02:52:48.824548 5052 ts_tablet_manager.cc:1397] T 95365a93b5554c37b6e7364b9ab4d03c P 9c31aad209d04d6eb97f15899d7972dd: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.004s
I20260806 02:52:48.824942 5053 log.cc:824] T 95365a93b5554c37b6e7364b9ab4d03c P 916e85b805ce4cbab170fdf74b250f9e: Log is configured to *not* fsync() on all Append() calls
I20260806 02:52:48.828099 5053 tablet_bootstrap.cc:492] T 95365a93b5554c37b6e7364b9ab4d03c P 916e85b805ce4cbab170fdf74b250f9e: No bootstrap required, opened a new log
I20260806 02:52:48.828380 5053 ts_tablet_manager.cc:1397] T 95365a93b5554c37b6e7364b9ab4d03c P 916e85b805ce4cbab170fdf74b250f9e: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260806 02:52:48.830647 5057 tablet_bootstrap.cc:492] T 95365a93b5554c37b6e7364b9ab4d03c P 72d6e066b33b4d3089f69ac997a038e2: Bootstrap starting.
I20260806 02:52:48.836176 5057 tablet_bootstrap.cc:654] T 95365a93b5554c37b6e7364b9ab4d03c P 72d6e066b33b4d3089f69ac997a038e2: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:48.836205 5052 raft_consensus.cc:348] T 95365a93b5554c37b6e7364b9ab4d03c P 9c31aad209d04d6eb97f15899d7972dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c31aad209d04d6eb97f15899d7972dd" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 44003 } } peers { permanent_uuid: "72d6e066b33b4d3089f69ac997a038e2" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 39997 } } peers { permanent_uuid: "916e85b805ce4cbab170fdf74b250f9e" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 42313 } }
I20260806 02:52:48.836750 5052 raft_consensus.cc:374] T 95365a93b5554c37b6e7364b9ab4d03c P 9c31aad209d04d6eb97f15899d7972dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:48.836930 5052 raft_consensus.cc:729] T 95365a93b5554c37b6e7364b9ab4d03c P 9c31aad209d04d6eb97f15899d7972dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c31aad209d04d6eb97f15899d7972dd, State: Initialized, Role: FOLLOWER
I20260806 02:52:48.837731 5052 consensus_queue.cc:260] T 95365a93b5554c37b6e7364b9ab4d03c P 9c31aad209d04d6eb97f15899d7972dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c31aad209d04d6eb97f15899d7972dd" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 44003 } } peers { permanent_uuid: "72d6e066b33b4d3089f69ac997a038e2" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 39997 } } peers { permanent_uuid: "916e85b805ce4cbab170fdf74b250f9e" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 42313 } }
I20260806 02:52:48.838078 5057 log.cc:824] T 95365a93b5554c37b6e7364b9ab4d03c P 72d6e066b33b4d3089f69ac997a038e2: Log is configured to *not* fsync() on all Append() calls
I20260806 02:52:48.842284 5053 raft_consensus.cc:348] T 95365a93b5554c37b6e7364b9ab4d03c P 916e85b805ce4cbab170fdf74b250f9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c31aad209d04d6eb97f15899d7972dd" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 44003 } } peers { permanent_uuid: "72d6e066b33b4d3089f69ac997a038e2" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 39997 } } peers { permanent_uuid: "916e85b805ce4cbab170fdf74b250f9e" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 42313 } }
I20260806 02:52:48.842869 5053 raft_consensus.cc:374] T 95365a93b5554c37b6e7364b9ab4d03c P 916e85b805ce4cbab170fdf74b250f9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:48.842978 5053 raft_consensus.cc:729] T 95365a93b5554c37b6e7364b9ab4d03c P 916e85b805ce4cbab170fdf74b250f9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 916e85b805ce4cbab170fdf74b250f9e, State: Initialized, Role: FOLLOWER
I20260806 02:52:48.843916 5053 consensus_queue.cc:260] T 95365a93b5554c37b6e7364b9ab4d03c P 916e85b805ce4cbab170fdf74b250f9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c31aad209d04d6eb97f15899d7972dd" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 44003 } } peers { permanent_uuid: "72d6e066b33b4d3089f69ac997a038e2" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 39997 } } peers { permanent_uuid: "916e85b805ce4cbab170fdf74b250f9e" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 42313 } }
I20260806 02:52:48.845229 5052 ts_tablet_manager.cc:1428] T 95365a93b5554c37b6e7364b9ab4d03c P 9c31aad209d04d6eb97f15899d7972dd: Time spent starting tablet: real 0.021s user 0.007s sys 0.010s
W20260806 02:52:48.850234 4573 tablet.cc:2369] T 95365a93b5554c37b6e7364b9ab4d03c P 9c31aad209d04d6eb97f15899d7972dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:52:48.851200 4782 heartbeater.cc:503] Master 127.25.211.190:45807 was elected leader, sending a full tablet report...
I20260806 02:52:48.851676 5053 ts_tablet_manager.cc:1428] T 95365a93b5554c37b6e7364b9ab4d03c P 916e85b805ce4cbab170fdf74b250f9e: Time spent starting tablet: real 0.023s user 0.015s sys 0.004s
I20260806 02:52:48.852542 5057 tablet_bootstrap.cc:492] T 95365a93b5554c37b6e7364b9ab4d03c P 72d6e066b33b4d3089f69ac997a038e2: No bootstrap required, opened a new log
I20260806 02:52:48.853046 5057 ts_tablet_manager.cc:1397] T 95365a93b5554c37b6e7364b9ab4d03c P 72d6e066b33b4d3089f69ac997a038e2: Time spent bootstrapping tablet: real 0.023s user 0.014s sys 0.007s
I20260806 02:52:48.863230 5057 raft_consensus.cc:348] T 95365a93b5554c37b6e7364b9ab4d03c P 72d6e066b33b4d3089f69ac997a038e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c31aad209d04d6eb97f15899d7972dd" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 44003 } } peers { permanent_uuid: "72d6e066b33b4d3089f69ac997a038e2" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 39997 } } peers { permanent_uuid: "916e85b805ce4cbab170fdf74b250f9e" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 42313 } }
I20260806 02:52:48.863839 5057 raft_consensus.cc:374] T 95365a93b5554c37b6e7364b9ab4d03c P 72d6e066b33b4d3089f69ac997a038e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:48.863977 5057 raft_consensus.cc:729] T 95365a93b5554c37b6e7364b9ab4d03c P 72d6e066b33b4d3089f69ac997a038e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72d6e066b33b4d3089f69ac997a038e2, State: Initialized, Role: FOLLOWER
I20260806 02:52:48.864648 5057 consensus_queue.cc:260] T 95365a93b5554c37b6e7364b9ab4d03c P 72d6e066b33b4d3089f69ac997a038e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c31aad209d04d6eb97f15899d7972dd" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 44003 } } peers { permanent_uuid: "72d6e066b33b4d3089f69ac997a038e2" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 39997 } } peers { permanent_uuid: "916e85b805ce4cbab170fdf74b250f9e" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 42313 } }
I20260806 02:52:48.866099 4996 heartbeater.cc:503] Master 127.25.211.190:45807 was elected leader, sending a full tablet report...
I20260806 02:52:48.867388 5057 ts_tablet_manager.cc:1428] T 95365a93b5554c37b6e7364b9ab4d03c P 72d6e066b33b4d3089f69ac997a038e2: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
W20260806 02:52:48.921607 5001 tablet.cc:2369] T 95365a93b5554c37b6e7364b9ab4d03c P 72d6e066b33b4d3089f69ac997a038e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:52:49.047407 4785 tablet.cc:2369] T 95365a93b5554c37b6e7364b9ab4d03c P 916e85b805ce4cbab170fdf74b250f9e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:52:49.074084 5066 raft_consensus.cc:482] T 95365a93b5554c37b6e7364b9ab4d03c P 72d6e066b33b4d3089f69ac997a038e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:52:49.074651 5066 raft_consensus.cc:504] T 95365a93b5554c37b6e7364b9ab4d03c P 72d6e066b33b4d3089f69ac997a038e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c31aad209d04d6eb97f15899d7972dd" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 44003 } } peers { permanent_uuid: "72d6e066b33b4d3089f69ac997a038e2" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 39997 } } peers { permanent_uuid: "916e85b805ce4cbab170fdf74b250f9e" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 42313 } }
I20260806 02:52:49.077037 5066 leader_election.cc:288] T 95365a93b5554c37b6e7364b9ab4d03c P 72d6e066b33b4d3089f69ac997a038e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c31aad209d04d6eb97f15899d7972dd (127.25.211.129:44003), 916e85b805ce4cbab170fdf74b250f9e (127.25.211.130:42313)
I20260806 02:52:49.084556 5048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:49.077854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:36299 (local address 127.25.211.130:42313)
0806 02:52:49.078087 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:49.078093 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:49.078115 (+ 22us) server_negotiation.cc:407] Connection header received
0806 02:52:49.078230 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:49.078235 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:49.078317 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:49.078417 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:52:49.079200 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:49.080062 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:49.082325 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:49.083436 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:49.083601 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:49.083680 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:49.084154 (+ 474us) server_negotiation.cc:299] Negotiation successful
0806 02:52:49.084288 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260806 02:52:49.084633 5010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:49.077397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.130:42313 (local address 127.25.211.131:36299)
0806 02:52:49.077901 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:49.077924 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:49.078073 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:49.078485 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:49.078488 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:49.078511 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:52:49.079039 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:49.079052 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:49.080242 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:49.080246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:49.082111 (+ 1865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:49.082125 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:49.083666 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:49.083671 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:49.084000 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:49.084025 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 02:52:49.084107 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 02:52:49.084284 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":4}
I20260806 02:52:49.084619 5044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:49.077448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:58033 (local address 127.25.211.129:44003)
0806 02:52:49.077736 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:49.077743 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:49.078355 (+ 612us) server_negotiation.cc:407] Connection header received
0806 02:52:49.078460 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:49.078464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:49.078537 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:49.078646 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:52:49.079149 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:49.080291 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:49.082462 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:49.083483 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:49.083666 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:49.083742 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:49.084154 (+ 412us) server_negotiation.cc:299] Negotiation successful
0806 02:52:49.084292 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 02:52:49.084637 5007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:49.077967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:44003 (local address 127.25.211.131:58033)
0806 02:52:49.078130 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:49.078147 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:49.078354 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:49.078718 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:49.078723 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:49.078752 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:52:49.079003 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:49.079014 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:49.080443 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:49.080446 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:49.082301 (+ 1855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:49.082314 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:49.083647 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:49.083653 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:49.083984 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:49.084012 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 02:52:49.084101 (+ 89us) client_negotiation.cc:240] Negotiation successful
0806 02:52:49.084298 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":83}
I20260806 02:52:49.085875 4723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95365a93b5554c37b6e7364b9ab4d03c" candidate_uuid: "72d6e066b33b4d3089f69ac997a038e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "916e85b805ce4cbab170fdf74b250f9e" is_pre_election: true
I20260806 02:52:49.086023 4511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95365a93b5554c37b6e7364b9ab4d03c" candidate_uuid: "72d6e066b33b4d3089f69ac997a038e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c31aad209d04d6eb97f15899d7972dd" is_pre_election: true
I20260806 02:52:49.086522 4723 raft_consensus.cc:2454] T 95365a93b5554c37b6e7364b9ab4d03c P 916e85b805ce4cbab170fdf74b250f9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72d6e066b33b4d3089f69ac997a038e2 in term 0.
I20260806 02:52:49.086761 4511 raft_consensus.cc:2454] T 95365a93b5554c37b6e7364b9ab4d03c P 9c31aad209d04d6eb97f15899d7972dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72d6e066b33b4d3089f69ac997a038e2 in term 0.
I20260806 02:52:49.087363 4826 leader_election.cc:302] T 95365a93b5554c37b6e7364b9ab4d03c P 72d6e066b33b4d3089f69ac997a038e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72d6e066b33b4d3089f69ac997a038e2, 916e85b805ce4cbab170fdf74b250f9e; no voters:
I20260806 02:52:49.087873 5066 raft_consensus.cc:2781] T 95365a93b5554c37b6e7364b9ab4d03c P 72d6e066b33b4d3089f69ac997a038e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:52:49.088011 5066 raft_consensus.cc:482] T 95365a93b5554c37b6e7364b9ab4d03c P 72d6e066b33b4d3089f69ac997a038e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:52:49.088166 5066 raft_consensus.cc:3037] T 95365a93b5554c37b6e7364b9ab4d03c P 72d6e066b33b4d3089f69ac997a038e2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:49.090942 5066 raft_consensus.cc:504] T 95365a93b5554c37b6e7364b9ab4d03c P 72d6e066b33b4d3089f69ac997a038e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c31aad209d04d6eb97f15899d7972dd" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 44003 } } peers { permanent_uuid: "72d6e066b33b4d3089f69ac997a038e2" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 39997 } } peers { permanent_uuid: "916e85b805ce4cbab170fdf74b250f9e" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 42313 } }
I20260806 02:52:49.091596 5066 leader_election.cc:288] T 95365a93b5554c37b6e7364b9ab4d03c P 72d6e066b33b4d3089f69ac997a038e2 [CANDIDATE]: Term 1 election: Requested vote from peers 9c31aad209d04d6eb97f15899d7972dd (127.25.211.129:44003), 916e85b805ce4cbab170fdf74b250f9e (127.25.211.130:42313)
I20260806 02:52:49.092214 4511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95365a93b5554c37b6e7364b9ab4d03c" candidate_uuid: "72d6e066b33b4d3089f69ac997a038e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c31aad209d04d6eb97f15899d7972dd"
I20260806 02:52:49.092284 4723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95365a93b5554c37b6e7364b9ab4d03c" candidate_uuid: "72d6e066b33b4d3089f69ac997a038e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "916e85b805ce4cbab170fdf74b250f9e"
I20260806 02:52:49.092514 4511 raft_consensus.cc:3037] T 95365a93b5554c37b6e7364b9ab4d03c P 9c31aad209d04d6eb97f15899d7972dd [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:49.092653 4723 raft_consensus.cc:3037] T 95365a93b5554c37b6e7364b9ab4d03c P 916e85b805ce4cbab170fdf74b250f9e [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:49.095559 4511 raft_consensus.cc:2454] T 95365a93b5554c37b6e7364b9ab4d03c P 9c31aad209d04d6eb97f15899d7972dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72d6e066b33b4d3089f69ac997a038e2 in term 1.
I20260806 02:52:49.095559 4723 raft_consensus.cc:2454] T 95365a93b5554c37b6e7364b9ab4d03c P 916e85b805ce4cbab170fdf74b250f9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72d6e066b33b4d3089f69ac997a038e2 in term 1.
I20260806 02:52:49.096149 4826 leader_election.cc:302] T 95365a93b5554c37b6e7364b9ab4d03c P 72d6e066b33b4d3089f69ac997a038e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 72d6e066b33b4d3089f69ac997a038e2, 916e85b805ce4cbab170fdf74b250f9e, 9c31aad209d04d6eb97f15899d7972dd; no voters:
I20260806 02:52:49.096542 5066 raft_consensus.cc:2781] T 95365a93b5554c37b6e7364b9ab4d03c P 72d6e066b33b4d3089f69ac997a038e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:52:49.097309 5066 raft_consensus.cc:686] T 95365a93b5554c37b6e7364b9ab4d03c P 72d6e066b33b4d3089f69ac997a038e2 [term 1 LEADER]: Becoming Leader. State: Replica: 72d6e066b33b4d3089f69ac997a038e2, State: Running, Role: LEADER
I20260806 02:52:49.097983 5066 consensus_queue.cc:237] T 95365a93b5554c37b6e7364b9ab4d03c P 72d6e066b33b4d3089f69ac997a038e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c31aad209d04d6eb97f15899d7972dd" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 44003 } } peers { permanent_uuid: "72d6e066b33b4d3089f69ac997a038e2" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 39997 } } peers { permanent_uuid: "916e85b805ce4cbab170fdf74b250f9e" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 42313 } }
I20260806 02:52:49.104206 3120 catalog_manager.cc:5495] T 95365a93b5554c37b6e7364b9ab4d03c P 72d6e066b33b4d3089f69ac997a038e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 72d6e066b33b4d3089f69ac997a038e2 (127.25.211.131). New cstate: current_term: 1 leader_uuid: "72d6e066b33b4d3089f69ac997a038e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c31aad209d04d6eb97f15899d7972dd" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 44003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72d6e066b33b4d3089f69ac997a038e2" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 39997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "916e85b805ce4cbab170fdf74b250f9e" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 42313 } health_report { overall_health: UNKNOWN } } }
I20260806 02:52:49.150650 5050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:49.144827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57578 (local address 127.25.211.131:39997)
0806 02:52:49.145027 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:49.145032 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:49.145192 (+ 160us) server_negotiation.cc:407] Connection header received
0806 02:52:49.145379 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:49.145382 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:49.145466 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:49.145607 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:52:49.145903 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:49.146713 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:49.148403 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:49.148822 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:49.148917 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:49.149454 (+ 537us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:52:49.150015 (+ 561us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:52:49.150126 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:49.150304 (+ 178us) server_negotiation.cc:299] Negotiation successful
0806 02:52:49.150379 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 02:52:49.161151 4723 raft_consensus.cc:1261] T 95365a93b5554c37b6e7364b9ab4d03c P 916e85b805ce4cbab170fdf74b250f9e [term 1 FOLLOWER]: Refusing update from remote peer 72d6e066b33b4d3089f69ac997a038e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:52:49.161177 4511 raft_consensus.cc:1261] T 95365a93b5554c37b6e7364b9ab4d03c P 9c31aad209d04d6eb97f15899d7972dd [term 1 FOLLOWER]: Refusing update from remote peer 72d6e066b33b4d3089f69ac997a038e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:52:49.162425 5067 consensus_queue.cc:1035] T 95365a93b5554c37b6e7364b9ab4d03c P 72d6e066b33b4d3089f69ac997a038e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c31aad209d04d6eb97f15899d7972dd" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 44003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:52:49.162847 5066 consensus_queue.cc:1035] T 95365a93b5554c37b6e7364b9ab4d03c P 72d6e066b33b4d3089f69ac997a038e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "916e85b805ce4cbab170fdf74b250f9e" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 42313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:52:49.181102 5070 mvcc.cc:205] Tried to move back new op lower bound from 7315393614472785920 to 7315393614229749760. Current Snapshot: MvccSnapshot[applied={T|T < 7315393614472785920}]
I20260806 02:52:49.201100 3120 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:52:49.201529 3120 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:52:49.209452 3120 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5259274cdfee46ca8a88c8a0a0723c80: Sending DeleteTablet for 3 replicas of tablet 95365a93b5554c37b6e7364b9ab4d03c
I20260806 02:52:49.210968 4484 tablet_service.cc:1514] Processing DeleteTablet for tablet 95365a93b5554c37b6e7364b9ab4d03c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:52:49 UTC) from {username='oryx', principal='oryx/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:39464
I20260806 02:52:49.211211 4892 tablet_service.cc:1514] Processing DeleteTablet for tablet 95365a93b5554c37b6e7364b9ab4d03c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:52:49 UTC) from {username='oryx', principal='oryx/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:57572
I20260806 02:52:49.211169 4692 tablet_service.cc:1514] Processing DeleteTablet for tablet 95365a93b5554c37b6e7364b9ab4d03c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:52:49 UTC) from {username='oryx', principal='oryx/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:34360
I20260806 02:52:49.212242 5085 tablet_replica.cc:330] T 95365a93b5554c37b6e7364b9ab4d03c P 9c31aad209d04d6eb97f15899d7972dd: stopping tablet replica
I20260806 02:52:49.212885 5085 raft_consensus.cc:2229] T 95365a93b5554c37b6e7364b9ab4d03c P 9c31aad209d04d6eb97f15899d7972dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:52:49.213577 5085 raft_consensus.cc:2258] T 95365a93b5554c37b6e7364b9ab4d03c P 9c31aad209d04d6eb97f15899d7972dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:52:49.215921 5085 ts_tablet_manager.cc:1905] T 95365a93b5554c37b6e7364b9ab4d03c P 9c31aad209d04d6eb97f15899d7972dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:52:49.218812 5086 tablet_replica.cc:330] T 95365a93b5554c37b6e7364b9ab4d03c P 72d6e066b33b4d3089f69ac997a038e2: stopping tablet replica
I20260806 02:52:49.219089 5087 tablet_replica.cc:330] T 95365a93b5554c37b6e7364b9ab4d03c P 916e85b805ce4cbab170fdf74b250f9e: stopping tablet replica
I20260806 02:52:49.219452 5086 raft_consensus.cc:2229] T 95365a93b5554c37b6e7364b9ab4d03c P 72d6e066b33b4d3089f69ac997a038e2 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:52:49.219776 5087 raft_consensus.cc:2229] T 95365a93b5554c37b6e7364b9ab4d03c P 916e85b805ce4cbab170fdf74b250f9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:52:49.220355 5087 raft_consensus.cc:2258] T 95365a93b5554c37b6e7364b9ab4d03c P 916e85b805ce4cbab170fdf74b250f9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:52:49.220436 5086 raft_consensus.cc:2258] T 95365a93b5554c37b6e7364b9ab4d03c P 72d6e066b33b4d3089f69ac997a038e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:52:49.223160 5086 ts_tablet_manager.cc:1905] T 95365a93b5554c37b6e7364b9ab4d03c P 72d6e066b33b4d3089f69ac997a038e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:52:49.228616 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 3070
I20260806 02:52:49.228785 5087 ts_tablet_manager.cc:1905] T 95365a93b5554c37b6e7364b9ab4d03c P 916e85b805ce4cbab170fdf74b250f9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:52:49.229785 3169 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:52:49.232197 5085 ts_tablet_manager.cc:1918] T 95365a93b5554c37b6e7364b9ab4d03c P 9c31aad209d04d6eb97f15899d7972dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:52:49.232465 5085 log.cc:1196] T 95365a93b5554c37b6e7364b9ab4d03c P 9c31aad209d04d6eb97f15899d7972dd: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-0/wal/wals/95365a93b5554c37b6e7364b9ab4d03c
I20260806 02:52:49.232911 5086 ts_tablet_manager.cc:1918] T 95365a93b5554c37b6e7364b9ab4d03c P 72d6e066b33b4d3089f69ac997a038e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:52:49.233253 5086 log.cc:1196] T 95365a93b5554c37b6e7364b9ab4d03c P 72d6e066b33b4d3089f69ac997a038e2: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-2/wal/wals/95365a93b5554c37b6e7364b9ab4d03c
I20260806 02:52:49.233439 5085 ts_tablet_manager.cc:1939] T 95365a93b5554c37b6e7364b9ab4d03c P 9c31aad209d04d6eb97f15899d7972dd: Deleting consensus metadata
I20260806 02:52:49.234171 5086 ts_tablet_manager.cc:1939] T 95365a93b5554c37b6e7364b9ab4d03c P 72d6e066b33b4d3089f69ac997a038e2: Deleting consensus metadata
I20260806 02:52:49.238595 5087 ts_tablet_manager.cc:1918] T 95365a93b5554c37b6e7364b9ab4d03c P 916e85b805ce4cbab170fdf74b250f9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:52:49.238797 5087 log.cc:1196] T 95365a93b5554c37b6e7364b9ab4d03c P 916e85b805ce4cbab170fdf74b250f9e: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-1/wal/wals/95365a93b5554c37b6e7364b9ab4d03c
I20260806 02:52:49.239683 5087 ts_tablet_manager.cc:1939] T 95365a93b5554c37b6e7364b9ab4d03c P 916e85b805ce4cbab170fdf74b250f9e: Deleting consensus metadata
I20260806 02:52:49.255134 5006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:49.219148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54116 (local address 127.25.211.189:44717)
0806 02:52:49.219333 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:49.219340 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:49.226764 (+ 7424us) server_negotiation.cc:407] Connection header received
0806 02:52:49.227154 (+ 390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:49.227161 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:49.227252 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:49.227331 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:49.236474 (+ 9143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:49.237790 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:49.240392 (+ 2602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:49.241121 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:49.241247 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:49.248315 (+ 7068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:49.248322 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:49.248325 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:49.248370 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:49.252149 (+ 3779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:49.252951 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:49.252956 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:49.252958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:49.253110 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:49.253545 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:49.253549 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:49.253551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:49.253871 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:49.254199 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:49.254527 (+ 328us) server_negotiation.cc:299] Negotiation successful
0806 02:52:49.254770 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 02:52:49.270540 5002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:49.227332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43552 (local address 127.25.211.188:40381)
0806 02:52:49.227519 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:49.227525 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:49.255382 (+ 27857us) server_negotiation.cc:407] Connection header received
0806 02:52:49.255995 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:49.256001 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:49.256092 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:49.256169 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:49.258392 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:49.259442 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:49.260944 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:49.261363 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:49.261485 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:49.264326 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:49.264332 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:49.264335 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:49.264373 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:49.267529 (+ 3156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:49.268282 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:49.268286 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:49.268288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:49.268468 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:49.268841 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:49.268844 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:49.268846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:49.269187 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:49.269452 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:49.269847 (+ 395us) server_negotiation.cc:299] Negotiation successful
0806 02:52:49.270118 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 02:52:49.390870 3094 catalog_manager.cc:4841] TS 72d6e066b33b4d3089f69ac997a038e2 (127.25.211.131:39997): tablet 95365a93b5554c37b6e7364b9ab4d03c (table test-table [id=9ddafece9326495daa29c991d3bb2a59]) successfully deleted
I20260806 02:52:49.391862 3093 catalog_manager.cc:4841] TS 916e85b805ce4cbab170fdf74b250f9e (127.25.211.130:42313): tablet 95365a93b5554c37b6e7364b9ab4d03c (table test-table [id=9ddafece9326495daa29c991d3bb2a59]) successfully deleted
W20260806 02:52:49.391814 3094 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.25.211.131:39997, user_credentials={real_user=slave}} blocked reactor thread for 154813us
I20260806 02:52:49.392479 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 3070
I20260806 02:52:49.444615 5006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:49.423986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54128 (local address 127.25.211.189:44717)
0806 02:52:49.424211 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:49.424217 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:49.425244 (+ 1027us) server_negotiation.cc:407] Connection header received
0806 02:52:49.425824 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:49.425831 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:49.425931 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:49.426012 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:49.428744 (+ 2732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:49.430032 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:49.432619 (+ 2587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:49.433285 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:49.433391 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:49.437465 (+ 4074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:49.437471 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:49.437474 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:49.437515 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:49.440852 (+ 3337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:49.441748 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:49.441754 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:49.441758 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:49.441956 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:49.442600 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:49.442606 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:49.442610 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:49.443016 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:49.443274 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:49.444020 (+ 746us) server_negotiation.cc:299] Negotiation successful
0806 02:52:49.444290 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 02:52:49.450304 5002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:49.425398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43558 (local address 127.25.211.188:40381)
0806 02:52:49.425654 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:49.425661 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:49.433123 (+ 7462us) server_negotiation.cc:407] Connection header received
0806 02:52:49.433742 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:49.433748 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:49.433860 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:49.433941 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:49.437118 (+ 3177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:49.438514 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:49.440370 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:49.441089 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:49.441236 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:49.444339 (+ 3103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:49.444346 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:49.444350 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:49.444394 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:49.447501 (+ 3107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:49.448204 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:49.448208 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:49.448210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:49.448349 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:49.448778 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:49.448782 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:49.448784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:49.449079 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:49.449308 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:49.449705 (+ 397us) server_negotiation.cc:299] Negotiation successful
0806 02:52:49.449947 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 02:52:50.190707 5100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:50.189716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45807 (local address 127.25.211.130:56169)
0806 02:52:50.190316 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:50.190489 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":251,"threads_started":1}
W20260806 02:52:50.191387 4782 heartbeater.cc:644] Failed to heartbeat to 127.25.211.190:45807 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.211.190:45807: connect: Connection refused (error 111)
I20260806 02:52:50.191605 5101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:50.190723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45807 (local address 127.25.211.131:49429)
0806 02:52:50.191271 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:50.191425 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":319,"threads_started":1}
W20260806 02:52:50.191984 4996 heartbeater.cc:644] Failed to heartbeat to 127.25.211.190:45807 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.211.190:45807: connect: Connection refused (error 111)
I20260806 02:52:50.197234 5102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:50.196268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45807 (local address 127.25.211.129:53365)
0806 02:52:50.196910 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:50.197048 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":322,"threads_started":1}
W20260806 02:52:50.197681 4569 heartbeater.cc:644] Failed to heartbeat to 127.25.211.190:45807 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.211.190:45807: connect: Connection refused (error 111)
I20260806 02:52:50.709599 5103 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5259274cdfee46ca8a88c8a0a0723c80)
I20260806 02:52:50.709892 5104 raft_consensus.cc:482] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5259274cdfee46ca8a88c8a0a0723c80)
I20260806 02:52:50.709942 5103 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5259274cdfee46ca8a88c8a0a0723c80" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45807 } } peers { permanent_uuid: "edad4ea4e4374f1383fb26cf1c316e03" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 44717 } attrs { promote: false } } peers { permanent_uuid: "f8a064e7081e4bc7b6da2b9c03c7e98c" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 40381 } attrs { promote: false } }
I20260806 02:52:50.710101 5104 raft_consensus.cc:504] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5259274cdfee46ca8a88c8a0a0723c80" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45807 } } peers { permanent_uuid: "edad4ea4e4374f1383fb26cf1c316e03" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 44717 } attrs { promote: false } } peers { permanent_uuid: "f8a064e7081e4bc7b6da2b9c03c7e98c" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 40381 } attrs { promote: false } }
I20260806 02:52:50.710939 5104 leader_election.cc:288] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5259274cdfee46ca8a88c8a0a0723c80 (127.25.211.190:45807), f8a064e7081e4bc7b6da2b9c03c7e98c (127.25.211.188:40381)
I20260806 02:52:50.712816 5106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:50.711688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45807 (local address 127.0.0.1:41626)
0806 02:52:50.712213 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:50.712452 (+ 239us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":226,"threads_started":1}
I20260806 02:52:50.713032 5105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:50.711203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45807 (local address 127.0.0.1:41622)
0806 02:52:50.712529 (+ 1326us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:50.712754 (+ 225us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1203,"thread_start_us":1056,"threads_started":1}
W20260806 02:52:50.713580 4287 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45807: connect: Connection refused (error 111)
W20260806 02:52:50.714350 3998 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45807: connect: Connection refused (error 111)
I20260806 02:52:50.714425 5103 leader_election.cc:288] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5259274cdfee46ca8a88c8a0a0723c80 (127.25.211.190:45807), edad4ea4e4374f1383fb26cf1c316e03 (127.25.211.189:44717)
I20260806 02:52:50.718597 5107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:50.717814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45807 (local address 127.0.0.1:41632)
0806 02:52:50.718008 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:50.718329 (+ 321us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260806 02:52:50.718935 4287 leader_election.cc:334] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5259274cdfee46ca8a88c8a0a0723c80 (127.25.211.190:45807): Network error: Client connection negotiation failed: client connection to 127.25.211.190:45807: connect: Connection refused (error 111)
I20260806 02:52:50.719209 5108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:50.718518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45807 (local address 127.0.0.1:41634)
0806 02:52:50.718707 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:50.718949 (+ 242us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260806 02:52:50.719507 3998 leader_election.cc:334] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5259274cdfee46ca8a88c8a0a0723c80 (127.25.211.190:45807): Network error: Client connection negotiation failed: client connection to 127.25.211.190:45807: connect: Connection refused (error 111)
I20260806 02:52:50.721796 5109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:50.712950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54142 (local address 127.25.211.189:44717)
0806 02:52:50.714031 (+ 1081us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:50.714037 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:50.714054 (+ 17us) server_negotiation.cc:407] Connection header received
0806 02:52:50.714120 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:50.714124 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:50.714229 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:50.714994 (+ 765us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:52:50.715221 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:50.716633 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:50.719399 (+ 2766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:50.720756 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:50.720933 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:50.721010 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:50.721434 (+ 424us) server_negotiation.cc:299] Negotiation successful
0806 02:52:50.721551 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":952,"thread_start_us":1262,"threads_started":1}
I20260806 02:52:50.721907 5106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:50.712549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.189:44717 (local address 127.0.0.1:54142)
0806 02:52:50.713365 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:50.713386 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:50.713530 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:50.714424 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:50.714430 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:50.714490 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:52:50.714813 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:50.714835 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:50.716838 (+ 2003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:50.716844 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:50.719159 (+ 2315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:50.719176 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:50.720995 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:50.721001 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:50.721231 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:50.721270 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 02:52:50.721381 (+ 111us) client_negotiation.cc:240] Negotiation successful
0806 02:52:50.721545 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"mutex_wait_us":6,"thread_start_us":583,"threads_started":1}
I20260806 02:52:50.722820 4033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f8a064e7081e4bc7b6da2b9c03c7e98c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "edad4ea4e4374f1383fb26cf1c316e03" is_pre_election: true
I20260806 02:52:50.723248 4033 raft_consensus.cc:2454] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8a064e7081e4bc7b6da2b9c03c7e98c in term 1.
I20260806 02:52:50.723613 5105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:50.712875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:40381 (local address 127.0.0.1:43570)
0806 02:52:50.713354 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:50.713378 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:50.713529 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:50.715398 (+ 1869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:50.715404 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:50.715439 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:52:50.715842 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:50.715854 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:50.718649 (+ 2795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:50.718654 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:50.720920 (+ 2266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:50.720933 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:50.722657 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:50.722664 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:50.722878 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:50.722914 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 02:52:50.723017 (+ 103us) client_negotiation.cc:240] Negotiation successful
0806 02:52:50.723180 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"mutex_wait_us":2,"thread_start_us":1155,"threads_started":1}
I20260806 02:52:50.723639 5112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:50.713829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43570 (local address 127.25.211.188:40381)
0806 02:52:50.715042 (+ 1213us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:50.715049 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:50.715078 (+ 29us) server_negotiation.cc:407] Connection header received
0806 02:52:50.715142 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:50.715147 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:50.715216 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:50.715989 (+ 773us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:52:50.716241 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:50.718476 (+ 2235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:50.721099 (+ 2623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:50.722410 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:50.722585 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:50.722638 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:50.723090 (+ 452us) server_negotiation.cc:299] Negotiation successful
0806 02:52:50.723199 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":734,"server-negotiator.queue_time_us":1083,"thread_start_us":952,"threads_started":1}
I20260806 02:52:50.723827 4291 leader_election.cc:302] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: edad4ea4e4374f1383fb26cf1c316e03, f8a064e7081e4bc7b6da2b9c03c7e98c; no voters: 5259274cdfee46ca8a88c8a0a0723c80
I20260806 02:52:50.724249 5103 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 02:52:50.724390 5103 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5259274cdfee46ca8a88c8a0a0723c80)
I20260806 02:52:50.724524 5103 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c [term 1 FOLLOWER]: Advancing to term 2
I20260806 02:52:50.724886 4333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "edad4ea4e4374f1383fb26cf1c316e03" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f8a064e7081e4bc7b6da2b9c03c7e98c" is_pre_election: true
I20260806 02:52:50.727058 5103 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5259274cdfee46ca8a88c8a0a0723c80" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45807 } } peers { permanent_uuid: "edad4ea4e4374f1383fb26cf1c316e03" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 44717 } attrs { promote: false } } peers { permanent_uuid: "f8a064e7081e4bc7b6da2b9c03c7e98c" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 40381 } attrs { promote: false } }
I20260806 02:52:50.727497 4333 raft_consensus.cc:2379] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate edad4ea4e4374f1383fb26cf1c316e03 in current term 2: Already voted for candidate f8a064e7081e4bc7b6da2b9c03c7e98c in this term.
I20260806 02:52:50.727777 5103 leader_election.cc:288] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c [CANDIDATE]: Term 2 election: Requested vote from peers 5259274cdfee46ca8a88c8a0a0723c80 (127.25.211.190:45807), edad4ea4e4374f1383fb26cf1c316e03 (127.25.211.189:44717)
I20260806 02:52:50.728395 5106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:50.727898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45807 (local address 127.0.0.1:41648)
0806 02:52:50.728120 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:50.728214 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260806 02:52:50.728443 3998 leader_election.cc:302] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: edad4ea4e4374f1383fb26cf1c316e03; no voters: 5259274cdfee46ca8a88c8a0a0723c80, f8a064e7081e4bc7b6da2b9c03c7e98c
I20260806 02:52:50.728410 4033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f8a064e7081e4bc7b6da2b9c03c7e98c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "edad4ea4e4374f1383fb26cf1c316e03"
I20260806 02:52:50.728668 4033 raft_consensus.cc:3037] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03 [term 1 FOLLOWER]: Advancing to term 2
I20260806 02:52:50.729490 5106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:50.729115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45807 (local address 127.0.0.1:41656)
0806 02:52:50.729275 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:50.729358 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260806 02:52:50.729640 4287 leader_election.cc:334] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5259274cdfee46ca8a88c8a0a0723c80 (127.25.211.190:45807): Network error: Client connection negotiation failed: client connection to 127.25.211.190:45807: connect: Connection refused (error 111)
I20260806 02:52:50.731195 4033 raft_consensus.cc:2454] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8a064e7081e4bc7b6da2b9c03c7e98c in term 2.
I20260806 02:52:50.731374 5104 raft_consensus.cc:2726] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 02:52:50.731737 4291 leader_election.cc:302] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: edad4ea4e4374f1383fb26cf1c316e03, f8a064e7081e4bc7b6da2b9c03c7e98c; no voters: 5259274cdfee46ca8a88c8a0a0723c80
I20260806 02:52:50.732059 5103 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c [term 2 FOLLOWER]: Leader election won for term 2
I20260806 02:52:50.732512 5103 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c [term 2 LEADER]: Becoming Leader. State: Replica: f8a064e7081e4bc7b6da2b9c03c7e98c, State: Running, Role: LEADER
I20260806 02:52:50.732894 5103 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5259274cdfee46ca8a88c8a0a0723c80" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45807 } } peers { permanent_uuid: "edad4ea4e4374f1383fb26cf1c316e03" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 44717 } attrs { promote: false } } peers { permanent_uuid: "f8a064e7081e4bc7b6da2b9c03c7e98c" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 40381 } attrs { promote: false } }
I20260806 02:52:50.735148 5119 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8a064e7081e4bc7b6da2b9c03c7e98c. Latest consensus state: current_term: 2 leader_uuid: "f8a064e7081e4bc7b6da2b9c03c7e98c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5259274cdfee46ca8a88c8a0a0723c80" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45807 } } peers { permanent_uuid: "edad4ea4e4374f1383fb26cf1c316e03" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 44717 } attrs { promote: false } } peers { permanent_uuid: "f8a064e7081e4bc7b6da2b9c03c7e98c" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 40381 } attrs { promote: false } } }
I20260806 02:52:50.735469 5119 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c [sys.catalog]: This master's current role is: LEADER
I20260806 02:52:50.736119 5121 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:52:50.741149 5121 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:52:50.742120 5121 catalog_manager.cc:1259] Loaded cluster ID: c7f486c5c0544604a757ab99b279479f
I20260806 02:52:50.742254 5121 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:52:50.743470 5121 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:52:50.744405 5121 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c: Loaded TSK: 0
I20260806 02:52:50.745361 5121 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:52:50.801349 4311 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43558:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:52:50.809327 4311 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:52:50.813939 5106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:50.813472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45807 (local address 127.0.0.1:41678)
0806 02:52:50.813664 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:50.813770 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260806 02:52:50.814210 4033 raft_consensus.cc:1261] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03 [term 2 FOLLOWER]: Refusing update from remote peer f8a064e7081e4bc7b6da2b9c03c7e98c: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260806 02:52:50.814963 5106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:50.814603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45807 (local address 127.0.0.1:41692)
0806 02:52:50.814750 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:50.814828 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260806 02:52:50.814991 5119 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c [LEADER]: Connected to new peer: Peer: permanent_uuid: "edad4ea4e4374f1383fb26cf1c316e03" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 44717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260806 02:52:50.815332 4287 consensus_peers.cc:487] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c -> Peer 5259274cdfee46ca8a88c8a0a0723c80 (127.25.211.190:45807): Couldn't send request to peer 5259274cdfee46ca8a88c8a0a0723c80. Status: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45807: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 02:52:50.818674 5104 sys_catalog.cc:455] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8a064e7081e4bc7b6da2b9c03c7e98c. Latest consensus state: current_term: 2 leader_uuid: "f8a064e7081e4bc7b6da2b9c03c7e98c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5259274cdfee46ca8a88c8a0a0723c80" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45807 } } peers { permanent_uuid: "edad4ea4e4374f1383fb26cf1c316e03" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 44717 } attrs { promote: false } } peers { permanent_uuid: "f8a064e7081e4bc7b6da2b9c03c7e98c" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 40381 } attrs { promote: false } } }
I20260806 02:52:50.818980 5104 sys_catalog.cc:458] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:52:50.821089 5119 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "f8a064e7081e4bc7b6da2b9c03c7e98c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5259274cdfee46ca8a88c8a0a0723c80" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45807 } } peers { permanent_uuid: "edad4ea4e4374f1383fb26cf1c316e03" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 44717 } attrs { promote: false } } peers { permanent_uuid: "f8a064e7081e4bc7b6da2b9c03c7e98c" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 40381 } attrs { promote: false } } }
I20260806 02:52:50.821408 5119 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c [sys.catalog]: This master's current role is: LEADER
I20260806 02:52:50.821830 5104 sys_catalog.cc:455] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "f8a064e7081e4bc7b6da2b9c03c7e98c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5259274cdfee46ca8a88c8a0a0723c80" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45807 } } peers { permanent_uuid: "edad4ea4e4374f1383fb26cf1c316e03" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 44717 } attrs { promote: false } } peers { permanent_uuid: "f8a064e7081e4bc7b6da2b9c03c7e98c" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 40381 } attrs { promote: false } } }
I20260806 02:52:50.822113 5104 sys_catalog.cc:458] T 00000000000000000000000000000000 P edad4ea4e4374f1383fb26cf1c316e03 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:52:50.838827 5132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:50.831617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34374 (local address 127.25.211.130:42313)
0806 02:52:50.832111 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:50.832116 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:50.832135 (+ 19us) server_negotiation.cc:407] Connection header received
0806 02:52:50.832202 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:50.832206 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:50.832287 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:50.832391 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:52:50.832958 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:50.833720 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:50.836382 (+ 2662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:50.837586 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:50.837760 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:50.837841 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:50.838321 (+ 480us) server_negotiation.cc:299] Negotiation successful
0806 02:52:50.838486 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":272,"threads_started":1}
I20260806 02:52:50.839298 5106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:50.830389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:44003 (local address 127.0.0.1:39472)
0806 02:52:50.830587 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:50.830607 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:50.830721 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:50.832135 (+ 1414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:50.832140 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:50.832171 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:52:50.832377 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:50.832390 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:50.834550 (+ 2160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:50.834555 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:50.836532 (+ 1977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:50.836544 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:50.837959 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:50.837966 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:50.838436 (+ 470us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:50.838483 (+ 47us) client_negotiation.cc:769] Sending connection context
0806 02:52:50.838564 (+ 81us) client_negotiation.cc:240] Negotiation successful
0806 02:52:50.838820 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":4}
I20260806 02:52:50.839721 5107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:50.831212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.130:42313 (local address 127.0.0.1:34374)
0806 02:52:50.831395 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:50.831419 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:50.831539 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:50.832439 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:50.832444 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:50.832475 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:52:50.832818 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:50.832831 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:50.833877 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:50.833883 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:50.836118 (+ 2235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:50.836140 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:50.837849 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:50.837856 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:50.838037 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:50.838064 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 02:52:50.838259 (+ 195us) client_negotiation.cc:240] Negotiation successful
0806 02:52:50.838482 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":83}
I20260806 02:52:50.840693 5128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:50.830489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39472 (local address 127.25.211.129:44003)
0806 02:52:50.831712 (+ 1223us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:50.831720 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:50.831741 (+ 21us) server_negotiation.cc:407] Connection header received
0806 02:52:50.831826 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:50.831832 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:50.831914 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:50.832070 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:52:50.833029 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:50.834318 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:50.836762 (+ 2444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:50.837651 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:50.840025 (+ 2374us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:50.840143 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:50.840207 (+ 64us) server_negotiation.cc:299] Negotiation successful
0806 02:52:50.840368 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1087,"thread_start_us":422,"threads_started":1}
I20260806 02:52:50.840628 4692 tablet_service.cc:1467] Processing CreateTablet for tablet 8d0b4b4d984b46239006223a45b80629 (DEFAULT_TABLE table=test-table [id=15f22c3d065a4c6ca707a242a416e860]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:52:50.841101 4692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d0b4b4d984b46239006223a45b80629. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:50.842965 4484 tablet_service.cc:1467] Processing CreateTablet for tablet 8d0b4b4d984b46239006223a45b80629 (DEFAULT_TABLE table=test-table [id=15f22c3d065a4c6ca707a242a416e860]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:52:50.843554 4484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d0b4b4d984b46239006223a45b80629. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:50.843883 5131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:50.831348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.131:39997 (local address 127.0.0.1:57592)
0806 02:52:50.834649 (+ 3301us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:50.834668 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:50.834821 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:50.835607 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:50.835614 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:50.835645 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:52:50.836087 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:50.836103 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:50.837481 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:50.837487 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:50.841178 (+ 3691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:50.841197 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:50.842983 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:50.842991 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:50.843180 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:50.843219 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 02:52:50.843352 (+ 133us) client_negotiation.cc:240] Negotiation successful
0806 02:52:50.843508 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3166,"thread_start_us":2959,"threads_started":1}
I20260806 02:52:50.844700 5130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:50.830896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57592 (local address 127.25.211.131:39997)
0806 02:52:50.831399 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:50.831405 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:50.834961 (+ 3556us) server_negotiation.cc:407] Connection header received
0806 02:52:50.835045 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:50.835050 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:50.835272 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:50.835381 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:52:50.836266 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:50.837319 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:50.841393 (+ 4074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:50.842688 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:50.842864 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:50.842941 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:50.844208 (+ 1267us) server_negotiation.cc:299] Negotiation successful
0806 02:52:50.844383 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":266,"threads_started":1}
I20260806 02:52:50.845547 4892 tablet_service.cc:1467] Processing CreateTablet for tablet 8d0b4b4d984b46239006223a45b80629 (DEFAULT_TABLE table=test-table [id=15f22c3d065a4c6ca707a242a416e860]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:52:50.846071 4892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d0b4b4d984b46239006223a45b80629. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:50.848320 5134 tablet_bootstrap.cc:492] T 8d0b4b4d984b46239006223a45b80629 P 916e85b805ce4cbab170fdf74b250f9e: Bootstrap starting.
I20260806 02:52:50.850368 5134 tablet_bootstrap.cc:654] T 8d0b4b4d984b46239006223a45b80629 P 916e85b805ce4cbab170fdf74b250f9e: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:50.850920 5135 tablet_bootstrap.cc:492] T 8d0b4b4d984b46239006223a45b80629 P 9c31aad209d04d6eb97f15899d7972dd: Bootstrap starting.
I20260806 02:52:50.852969 5135 tablet_bootstrap.cc:654] T 8d0b4b4d984b46239006223a45b80629 P 9c31aad209d04d6eb97f15899d7972dd: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:50.855743 5135 tablet_bootstrap.cc:492] T 8d0b4b4d984b46239006223a45b80629 P 9c31aad209d04d6eb97f15899d7972dd: No bootstrap required, opened a new log
I20260806 02:52:50.855937 5135 ts_tablet_manager.cc:1397] T 8d0b4b4d984b46239006223a45b80629 P 9c31aad209d04d6eb97f15899d7972dd: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 02:52:50.856901 5135 raft_consensus.cc:348] T 8d0b4b4d984b46239006223a45b80629 P 9c31aad209d04d6eb97f15899d7972dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c31aad209d04d6eb97f15899d7972dd" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 44003 } } peers { permanent_uuid: "72d6e066b33b4d3089f69ac997a038e2" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 39997 } } peers { permanent_uuid: "916e85b805ce4cbab170fdf74b250f9e" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 42313 } }
I20260806 02:52:50.857197 5135 raft_consensus.cc:374] T 8d0b4b4d984b46239006223a45b80629 P 9c31aad209d04d6eb97f15899d7972dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:50.857300 5135 raft_consensus.cc:729] T 8d0b4b4d984b46239006223a45b80629 P 9c31aad209d04d6eb97f15899d7972dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c31aad209d04d6eb97f15899d7972dd, State: Initialized, Role: FOLLOWER
I20260806 02:52:50.857667 5135 consensus_queue.cc:260] T 8d0b4b4d984b46239006223a45b80629 P 9c31aad209d04d6eb97f15899d7972dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c31aad209d04d6eb97f15899d7972dd" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 44003 } } peers { permanent_uuid: "72d6e066b33b4d3089f69ac997a038e2" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 39997 } } peers { permanent_uuid: "916e85b805ce4cbab170fdf74b250f9e" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 42313 } }
I20260806 02:52:50.858791 5142 tablet_bootstrap.cc:492] T 8d0b4b4d984b46239006223a45b80629 P 72d6e066b33b4d3089f69ac997a038e2: Bootstrap starting.
I20260806 02:52:50.861081 5142 tablet_bootstrap.cc:654] T 8d0b4b4d984b46239006223a45b80629 P 72d6e066b33b4d3089f69ac997a038e2: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:50.864238 5134 tablet_bootstrap.cc:492] T 8d0b4b4d984b46239006223a45b80629 P 916e85b805ce4cbab170fdf74b250f9e: No bootstrap required, opened a new log
I20260806 02:52:50.864413 5134 ts_tablet_manager.cc:1397] T 8d0b4b4d984b46239006223a45b80629 P 916e85b805ce4cbab170fdf74b250f9e: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.011s
I20260806 02:52:50.864898 5135 ts_tablet_manager.cc:1428] T 8d0b4b4d984b46239006223a45b80629 P 9c31aad209d04d6eb97f15899d7972dd: Time spent starting tablet: real 0.009s user 0.005s sys 0.004s
I20260806 02:52:50.865190 5134 raft_consensus.cc:348] T 8d0b4b4d984b46239006223a45b80629 P 916e85b805ce4cbab170fdf74b250f9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c31aad209d04d6eb97f15899d7972dd" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 44003 } } peers { permanent_uuid: "72d6e066b33b4d3089f69ac997a038e2" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 39997 } } peers { permanent_uuid: "916e85b805ce4cbab170fdf74b250f9e" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 42313 } }
I20260806 02:52:50.865469 5134 raft_consensus.cc:374] T 8d0b4b4d984b46239006223a45b80629 P 916e85b805ce4cbab170fdf74b250f9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:50.865592 5134 raft_consensus.cc:729] T 8d0b4b4d984b46239006223a45b80629 P 916e85b805ce4cbab170fdf74b250f9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 916e85b805ce4cbab170fdf74b250f9e, State: Initialized, Role: FOLLOWER
I20260806 02:52:50.865661 5142 tablet_bootstrap.cc:492] T 8d0b4b4d984b46239006223a45b80629 P 72d6e066b33b4d3089f69ac997a038e2: No bootstrap required, opened a new log
I20260806 02:52:50.865896 5142 ts_tablet_manager.cc:1397] T 8d0b4b4d984b46239006223a45b80629 P 72d6e066b33b4d3089f69ac997a038e2: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260806 02:52:50.865869 5134 consensus_queue.cc:260] T 8d0b4b4d984b46239006223a45b80629 P 916e85b805ce4cbab170fdf74b250f9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c31aad209d04d6eb97f15899d7972dd" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 44003 } } peers { permanent_uuid: "72d6e066b33b4d3089f69ac997a038e2" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 39997 } } peers { permanent_uuid: "916e85b805ce4cbab170fdf74b250f9e" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 42313 } }
I20260806 02:52:50.866806 5142 raft_consensus.cc:348] T 8d0b4b4d984b46239006223a45b80629 P 72d6e066b33b4d3089f69ac997a038e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c31aad209d04d6eb97f15899d7972dd" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 44003 } } peers { permanent_uuid: "72d6e066b33b4d3089f69ac997a038e2" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 39997 } } peers { permanent_uuid: "916e85b805ce4cbab170fdf74b250f9e" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 42313 } }
I20260806 02:52:50.867058 5142 raft_consensus.cc:374] T 8d0b4b4d984b46239006223a45b80629 P 72d6e066b33b4d3089f69ac997a038e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:50.867163 5142 raft_consensus.cc:729] T 8d0b4b4d984b46239006223a45b80629 P 72d6e066b33b4d3089f69ac997a038e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72d6e066b33b4d3089f69ac997a038e2, State: Initialized, Role: FOLLOWER
I20260806 02:52:50.867419 5142 consensus_queue.cc:260] T 8d0b4b4d984b46239006223a45b80629 P 72d6e066b33b4d3089f69ac997a038e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c31aad209d04d6eb97f15899d7972dd" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 44003 } } peers { permanent_uuid: "72d6e066b33b4d3089f69ac997a038e2" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 39997 } } peers { permanent_uuid: "916e85b805ce4cbab170fdf74b250f9e" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 42313 } }
I20260806 02:52:50.870039 5152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:50.865661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45807 (local address 127.25.211.129:49269)
0806 02:52:50.869533 (+ 3872us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:50.869734 (+ 201us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3716,"threads_started":1}
I20260806 02:52:50.871205 4568 heartbeater.cc:503] Master 127.25.211.188:40381 was elected leader, sending a full tablet report...
I20260806 02:52:50.871469 5142 ts_tablet_manager.cc:1428] T 8d0b4b4d984b46239006223a45b80629 P 72d6e066b33b4d3089f69ac997a038e2: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260806 02:52:50.874682 5134 ts_tablet_manager.cc:1428] T 8d0b4b4d984b46239006223a45b80629 P 916e85b805ce4cbab170fdf74b250f9e: Time spent starting tablet: real 0.010s user 0.004s sys 0.005s
I20260806 02:52:50.875614 4780 heartbeater.cc:503] Master 127.25.211.188:40381 was elected leader, sending a full tablet report...
I20260806 02:52:50.876101 5157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:50.872152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45807 (local address 127.25.211.131:35447)
0806 02:52:50.875521 (+ 3369us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:50.875801 (+ 280us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3215,"thread_start_us":3321,"threads_started":1}
I20260806 02:52:50.877313 4991 heartbeater.cc:503] Master 127.25.211.188:40381 was elected leader, sending a full tablet report...
I20260806 02:52:50.878374 5157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:50.877899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45807 (local address 127.25.211.131:46145)
0806 02:52:50.878078 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:50.878216 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260806 02:52:50.887027 5158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:50.875333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45807 (local address 127.25.211.130:32897)
0806 02:52:50.886625 (+ 11292us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:50.886797 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11145,"threads_started":1}
I20260806 02:52:50.888628 5158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:50.888152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45807 (local address 127.25.211.130:33991)
0806 02:52:50.888350 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:50.888454 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260806 02:52:51.020767 5153 raft_consensus.cc:482] T 8d0b4b4d984b46239006223a45b80629 P 916e85b805ce4cbab170fdf74b250f9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:52:51.021026 5153 raft_consensus.cc:504] T 8d0b4b4d984b46239006223a45b80629 P 916e85b805ce4cbab170fdf74b250f9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c31aad209d04d6eb97f15899d7972dd" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 44003 } } peers { permanent_uuid: "72d6e066b33b4d3089f69ac997a038e2" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 39997 } } peers { permanent_uuid: "916e85b805ce4cbab170fdf74b250f9e" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 42313 } }
I20260806 02:52:51.022728 5153 leader_election.cc:288] T 8d0b4b4d984b46239006223a45b80629 P 916e85b805ce4cbab170fdf74b250f9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c31aad209d04d6eb97f15899d7972dd (127.25.211.129:44003), 72d6e066b33b4d3089f69ac997a038e2 (127.25.211.131:39997)
I20260806 02:52:51.029896 5158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:51.023065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:44003 (local address 127.25.211.130:48897)
0806 02:52:51.023329 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:51.023348 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:51.023486 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:51.024018 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:51.024026 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:51.024083 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:52:51.024369 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:51.024379 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:51.025684 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:51.025690 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:51.027532 (+ 1842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:51.027546 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:51.029034 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:51.029041 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:51.029237 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:51.029276 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 02:52:51.029388 (+ 112us) client_negotiation.cc:240] Negotiation successful
0806 02:52:51.029562 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":5}
I20260806 02:52:51.029744 5128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:51.023171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:48897 (local address 127.25.211.129:44003)
0806 02:52:51.023430 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:51.023437 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:51.023461 (+ 24us) server_negotiation.cc:407] Connection header received
0806 02:52:51.023639 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:51.023645 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:51.023737 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:51.023839 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:52:51.024523 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:51.025506 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:51.027704 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:51.028815 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:51.028941 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:51.028998 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:51.029428 (+ 430us) server_negotiation.cc:299] Negotiation successful
0806 02:52:51.029538 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 02:52:51.030947 5130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:51.023712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:49745 (local address 127.25.211.131:39997)
0806 02:52:51.023918 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:51.023924 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:51.024314 (+ 390us) server_negotiation.cc:407] Connection header received
0806 02:52:51.024562 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:51.024567 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:51.024629 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:51.024696 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:52:51.025322 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:51.026401 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:51.028578 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:51.029711 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:51.029869 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:51.029941 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:51.030510 (+ 569us) server_negotiation.cc:299] Negotiation successful
0806 02:52:51.030668 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 02:52:51.030849 4511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d0b4b4d984b46239006223a45b80629" candidate_uuid: "916e85b805ce4cbab170fdf74b250f9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c31aad209d04d6eb97f15899d7972dd" is_pre_election: true
I20260806 02:52:51.031005 5159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:51.023544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.131:39997 (local address 127.25.211.130:49745)
0806 02:52:51.024195 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:51.024210 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:51.024412 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:51.024737 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:51.024742 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:51.024769 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:52:51.025123 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:51.025137 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:51.026541 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:51.026545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:51.028380 (+ 1835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:51.028396 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:51.030079 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:51.030090 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:51.030330 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:51.030361 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 02:52:51.030461 (+ 100us) client_negotiation.cc:240] Negotiation successful
0806 02:52:51.030598 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"mutex_wait_us":23,"thread_start_us":363,"threads_started":1}
I20260806 02:52:51.031134 4511 raft_consensus.cc:2454] T 8d0b4b4d984b46239006223a45b80629 P 9c31aad209d04d6eb97f15899d7972dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 916e85b805ce4cbab170fdf74b250f9e in term 0.
I20260806 02:52:51.031685 4626 leader_election.cc:302] T 8d0b4b4d984b46239006223a45b80629 P 916e85b805ce4cbab170fdf74b250f9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 916e85b805ce4cbab170fdf74b250f9e, 9c31aad209d04d6eb97f15899d7972dd; no voters:
I20260806 02:52:51.031849 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d0b4b4d984b46239006223a45b80629" candidate_uuid: "916e85b805ce4cbab170fdf74b250f9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72d6e066b33b4d3089f69ac997a038e2" is_pre_election: true
I20260806 02:52:51.032081 5153 raft_consensus.cc:2781] T 8d0b4b4d984b46239006223a45b80629 P 916e85b805ce4cbab170fdf74b250f9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:52:51.032197 5153 raft_consensus.cc:482] T 8d0b4b4d984b46239006223a45b80629 P 916e85b805ce4cbab170fdf74b250f9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:52:51.032294 5153 raft_consensus.cc:3037] T 8d0b4b4d984b46239006223a45b80629 P 916e85b805ce4cbab170fdf74b250f9e [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:51.032267 4927 raft_consensus.cc:2454] T 8d0b4b4d984b46239006223a45b80629 P 72d6e066b33b4d3089f69ac997a038e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 916e85b805ce4cbab170fdf74b250f9e in term 0.
I20260806 02:52:51.034771 5153 raft_consensus.cc:504] T 8d0b4b4d984b46239006223a45b80629 P 916e85b805ce4cbab170fdf74b250f9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c31aad209d04d6eb97f15899d7972dd" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 44003 } } peers { permanent_uuid: "72d6e066b33b4d3089f69ac997a038e2" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 39997 } } peers { permanent_uuid: "916e85b805ce4cbab170fdf74b250f9e" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 42313 } }
I20260806 02:52:51.035416 5153 leader_election.cc:288] T 8d0b4b4d984b46239006223a45b80629 P 916e85b805ce4cbab170fdf74b250f9e [CANDIDATE]: Term 1 election: Requested vote from peers 9c31aad209d04d6eb97f15899d7972dd (127.25.211.129:44003), 72d6e066b33b4d3089f69ac997a038e2 (127.25.211.131:39997)
I20260806 02:52:51.035943 4927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d0b4b4d984b46239006223a45b80629" candidate_uuid: "916e85b805ce4cbab170fdf74b250f9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72d6e066b33b4d3089f69ac997a038e2"
I20260806 02:52:51.036031 4511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d0b4b4d984b46239006223a45b80629" candidate_uuid: "916e85b805ce4cbab170fdf74b250f9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c31aad209d04d6eb97f15899d7972dd"
I20260806 02:52:51.036139 4927 raft_consensus.cc:3037] T 8d0b4b4d984b46239006223a45b80629 P 72d6e066b33b4d3089f69ac997a038e2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:51.036273 4511 raft_consensus.cc:3037] T 8d0b4b4d984b46239006223a45b80629 P 9c31aad209d04d6eb97f15899d7972dd [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:51.038401 4511 raft_consensus.cc:2454] T 8d0b4b4d984b46239006223a45b80629 P 9c31aad209d04d6eb97f15899d7972dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 916e85b805ce4cbab170fdf74b250f9e in term 1.
I20260806 02:52:51.038443 4927 raft_consensus.cc:2454] T 8d0b4b4d984b46239006223a45b80629 P 72d6e066b33b4d3089f69ac997a038e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 916e85b805ce4cbab170fdf74b250f9e in term 1.
I20260806 02:52:51.038830 4626 leader_election.cc:302] T 8d0b4b4d984b46239006223a45b80629 P 916e85b805ce4cbab170fdf74b250f9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 916e85b805ce4cbab170fdf74b250f9e, 9c31aad209d04d6eb97f15899d7972dd; no voters:
I20260806 02:52:51.039196 5153 raft_consensus.cc:2781] T 8d0b4b4d984b46239006223a45b80629 P 916e85b805ce4cbab170fdf74b250f9e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:52:51.039738 5153 raft_consensus.cc:686] T 8d0b4b4d984b46239006223a45b80629 P 916e85b805ce4cbab170fdf74b250f9e [term 1 LEADER]: Becoming Leader. State: Replica: 916e85b805ce4cbab170fdf74b250f9e, State: Running, Role: LEADER
I20260806 02:52:51.040190 5153 consensus_queue.cc:237] T 8d0b4b4d984b46239006223a45b80629 P 916e85b805ce4cbab170fdf74b250f9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c31aad209d04d6eb97f15899d7972dd" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 44003 } } peers { permanent_uuid: "72d6e066b33b4d3089f69ac997a038e2" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 39997 } } peers { permanent_uuid: "916e85b805ce4cbab170fdf74b250f9e" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 42313 } }
I20260806 02:52:51.040982 5159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:51.040591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45807 (local address 127.25.211.130:42131)
0806 02:52:51.040756 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:51.040845 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260806 02:52:51.043908 4312 catalog_manager.cc:5495] T 8d0b4b4d984b46239006223a45b80629 P 916e85b805ce4cbab170fdf74b250f9e reported cstate change: term changed from 0 to 1, leader changed from <none> to 916e85b805ce4cbab170fdf74b250f9e (127.25.211.130). New cstate: current_term: 1 leader_uuid: "916e85b805ce4cbab170fdf74b250f9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c31aad209d04d6eb97f15899d7972dd" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 44003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72d6e066b33b4d3089f69ac997a038e2" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 39997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "916e85b805ce4cbab170fdf74b250f9e" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 42313 } health_report { overall_health: HEALTHY } } }
I20260806 02:52:51.092902 5132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:51.088145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34386 (local address 127.25.211.130:42313)
0806 02:52:51.088352 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:51.088357 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:51.088374 (+ 17us) server_negotiation.cc:407] Connection header received
0806 02:52:51.088466 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:51.088469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:51.088546 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:51.088667 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:52:51.089026 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:51.089759 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:51.091032 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:51.091450 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:51.091539 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:51.091897 (+ 358us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:52:51.092435 (+ 538us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:52:51.092538 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:51.092611 (+ 73us) server_negotiation.cc:299] Negotiation successful
0806 02:52:51.092676 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 02:52:51.097465 4511 raft_consensus.cc:1261] T 8d0b4b4d984b46239006223a45b80629 P 9c31aad209d04d6eb97f15899d7972dd [term 1 FOLLOWER]: Refusing update from remote peer 916e85b805ce4cbab170fdf74b250f9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:52:51.097767 4927 raft_consensus.cc:1261] T 8d0b4b4d984b46239006223a45b80629 P 72d6e066b33b4d3089f69ac997a038e2 [term 1 FOLLOWER]: Refusing update from remote peer 916e85b805ce4cbab170fdf74b250f9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:52:51.098326 5160 consensus_queue.cc:1035] T 8d0b4b4d984b46239006223a45b80629 P 916e85b805ce4cbab170fdf74b250f9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c31aad209d04d6eb97f15899d7972dd" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 44003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:52:51.098727 5153 consensus_queue.cc:1035] T 8d0b4b4d984b46239006223a45b80629 P 916e85b805ce4cbab170fdf74b250f9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "72d6e066b33b4d3089f69ac997a038e2" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:52:51.101238 5152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:51.100788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45807 (local address 127.25.211.129:47263)
0806 02:52:51.100935 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:51.101027 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260806 02:52:51.101676 5157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:51.101167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45807 (local address 127.25.211.131:49877)
0806 02:52:51.101398 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:51.101510 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260806 02:52:51.109285 5157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:51.108604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45807 (local address 127.25.211.131:39981)
0806 02:52:51.108970 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:51.109076 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246}
I20260806 02:52:51.109606 5152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:51.109034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45807 (local address 127.25.211.129:60285)
0806 02:52:51.109263 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:51.109357 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260806 02:52:51.118525 5159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:51.117975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45807 (local address 127.25.211.130:55623)
0806 02:52:51.118217 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:51.118341 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260806 02:52:51.127942 4311 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:52:51.128232 4311 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:52:51.134106 4311 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f8a064e7081e4bc7b6da2b9c03c7e98c: Sending DeleteTablet for 3 replicas of tablet 8d0b4b4d984b46239006223a45b80629
I20260806 02:52:51.135577 4692 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d0b4b4d984b46239006223a45b80629 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:52:51 UTC) from {username='oryx', principal='oryx/127.25.211.188@KRBTEST.COM'} at 127.0.0.1:34374
I20260806 02:52:51.136093 4892 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d0b4b4d984b46239006223a45b80629 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:52:51 UTC) from {username='oryx', principal='oryx/127.25.211.188@KRBTEST.COM'} at 127.0.0.1:57592
I20260806 02:52:51.136137 4484 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d0b4b4d984b46239006223a45b80629 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:52:51 UTC) from {username='oryx', principal='oryx/127.25.211.188@KRBTEST.COM'} at 127.0.0.1:39472
I20260806 02:52:51.138134 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 4366
I20260806 02:52:51.142612 5180 tablet_replica.cc:330] T 8d0b4b4d984b46239006223a45b80629 P 916e85b805ce4cbab170fdf74b250f9e: stopping tablet replica
I20260806 02:52:51.142866 5180 raft_consensus.cc:2229] T 8d0b4b4d984b46239006223a45b80629 P 916e85b805ce4cbab170fdf74b250f9e [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:52:51.146693 5182 tablet_replica.cc:330] T 8d0b4b4d984b46239006223a45b80629 P 9c31aad209d04d6eb97f15899d7972dd: stopping tablet replica
I20260806 02:52:51.147297 5181 tablet_replica.cc:330] T 8d0b4b4d984b46239006223a45b80629 P 72d6e066b33b4d3089f69ac997a038e2: stopping tablet replica
I20260806 02:52:51.147491 5181 raft_consensus.cc:2229] T 8d0b4b4d984b46239006223a45b80629 P 72d6e066b33b4d3089f69ac997a038e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:52:51.147781 5181 raft_consensus.cc:2258] T 8d0b4b4d984b46239006223a45b80629 P 72d6e066b33b4d3089f69ac997a038e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:52:51.148113 5182 raft_consensus.cc:2229] T 8d0b4b4d984b46239006223a45b80629 P 9c31aad209d04d6eb97f15899d7972dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:52:51.148432 5182 raft_consensus.cc:2258] T 8d0b4b4d984b46239006223a45b80629 P 9c31aad209d04d6eb97f15899d7972dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:52:51.149590 5181 ts_tablet_manager.cc:1905] T 8d0b4b4d984b46239006223a45b80629 P 72d6e066b33b4d3089f69ac997a038e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:52:51.149592 5182 ts_tablet_manager.cc:1905] T 8d0b4b4d984b46239006223a45b80629 P 9c31aad209d04d6eb97f15899d7972dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:52:51.154268 5180 raft_consensus.cc:2258] T 8d0b4b4d984b46239006223a45b80629 P 916e85b805ce4cbab170fdf74b250f9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:52:51.155471 5180 ts_tablet_manager.cc:1905] T 8d0b4b4d984b46239006223a45b80629 P 916e85b805ce4cbab170fdf74b250f9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:52:51.156523 5181 ts_tablet_manager.cc:1918] T 8d0b4b4d984b46239006223a45b80629 P 72d6e066b33b4d3089f69ac997a038e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:52:51.156669 5181 log.cc:1196] T 8d0b4b4d984b46239006223a45b80629 P 72d6e066b33b4d3089f69ac997a038e2: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-2/wal/wals/8d0b4b4d984b46239006223a45b80629
I20260806 02:52:51.157132 5181 ts_tablet_manager.cc:1939] T 8d0b4b4d984b46239006223a45b80629 P 72d6e066b33b4d3089f69ac997a038e2: Deleting consensus metadata
I20260806 02:52:51.157684 5182 ts_tablet_manager.cc:1918] T 8d0b4b4d984b46239006223a45b80629 P 9c31aad209d04d6eb97f15899d7972dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:52:51.157902 5182 log.cc:1196] T 8d0b4b4d984b46239006223a45b80629 P 9c31aad209d04d6eb97f15899d7972dd: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-0/wal/wals/8d0b4b4d984b46239006223a45b80629
I20260806 02:52:51.159052 5182 ts_tablet_manager.cc:1939] T 8d0b4b4d984b46239006223a45b80629 P 9c31aad209d04d6eb97f15899d7972dd: Deleting consensus metadata
I20260806 02:52:51.159019 4288 catalog_manager.cc:4841] TS 72d6e066b33b4d3089f69ac997a038e2 (127.25.211.131:39997): tablet 8d0b4b4d984b46239006223a45b80629 (table test-table [id=15f22c3d065a4c6ca707a242a416e860]) successfully deleted
I20260806 02:52:51.160353 4288 catalog_manager.cc:4841] TS 9c31aad209d04d6eb97f15899d7972dd (127.25.211.129:44003): tablet 8d0b4b4d984b46239006223a45b80629 (table test-table [id=15f22c3d065a4c6ca707a242a416e860]) successfully deleted
I20260806 02:52:51.162679 5180 ts_tablet_manager.cc:1918] T 8d0b4b4d984b46239006223a45b80629 P 916e85b805ce4cbab170fdf74b250f9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:52:51.162846 5180 log.cc:1196] T 8d0b4b4d984b46239006223a45b80629 P 916e85b805ce4cbab170fdf74b250f9e: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1785984707115702-26446-0/minicluster-data/ts-1/wal/wals/8d0b4b4d984b46239006223a45b80629
I20260806 02:52:51.163359 5180 ts_tablet_manager.cc:1939] T 8d0b4b4d984b46239006223a45b80629 P 916e85b805ce4cbab170fdf74b250f9e: Deleting consensus metadata
I20260806 02:52:51.164731 4287 catalog_manager.cc:4841] TS 916e85b805ce4cbab170fdf74b250f9e (127.25.211.130:42313): tablet 8d0b4b4d984b46239006223a45b80629 (table test-table [id=15f22c3d065a4c6ca707a242a416e860]) successfully deleted
I20260806 02:52:51.169992 5128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:51.139774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39480 (local address 127.25.211.129:44003)
0806 02:52:51.139973 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:51.139980 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:51.152481 (+ 12501us) server_negotiation.cc:407] Connection header received
0806 02:52:51.153026 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:51.153033 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:51.153262 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:51.153382 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:51.155584 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:51.156771 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:51.159065 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:51.159659 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:51.159767 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:51.163543 (+ 3776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:51.163566 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:51.163589 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:51.163636 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:51.166970 (+ 3334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:51.167760 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:51.167768 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:51.167771 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:51.167924 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:51.168343 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:51.168347 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:51.168350 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:51.168745 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:51.169071 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:51.169388 (+ 317us) server_negotiation.cc:299] Negotiation successful
0806 02:52:51.169655 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 02:52:51.170717 4563 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:52:51.234266 5131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:51.233779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45807 (local address 127.0.0.1:41706)
0806 02:52:51.233962 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:51.234066 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260806 02:52:51.235471 5131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:51.234966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45807 (local address 127.0.0.1:41718)
0806 02:52:51.235244 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:51.235343 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
I20260806 02:52:51.320750 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 4366
I20260806 02:52:51.346002 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 4588
I20260806 02:52:51.361009 5132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:51.346833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34400 (local address 127.25.211.130:42313)
0806 02:52:51.347362 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:51.347367 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:51.347384 (+ 17us) server_negotiation.cc:407] Connection header received
0806 02:52:51.347438 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:51.347442 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:51.347522 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:51.347617 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:51.348958 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:51.349777 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:51.351256 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:51.351739 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:51.351864 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:51.354616 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:51.354633 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:51.354650 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:51.354691 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:51.357857 (+ 3166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:51.358764 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:51.358772 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:51.358775 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:51.358934 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:51.359354 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:51.359358 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:51.359360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:51.359772 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:51.360103 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:51.360417 (+ 314us) server_negotiation.cc:299] Negotiation successful
0806 02:52:51.360655 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418}
I20260806 02:52:51.361821 4773 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:52:51.480396 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 4588
I20260806 02:52:51.506472 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 4801
I20260806 02:52:51.520655 5130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:51.507457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57604 (local address 127.25.211.131:39997)
0806 02:52:51.507639 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:51.507644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:51.507663 (+ 19us) server_negotiation.cc:407] Connection header received
0806 02:52:51.507758 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:51.507762 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:51.507834 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:51.507916 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:51.509209 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:51.509940 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:51.511360 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:51.511870 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:51.511984 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:51.514730 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:51.514744 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:51.514762 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:51.514797 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:51.517650 (+ 2853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:51.518509 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:51.518518 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:51.518521 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:51.518692 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:51.519039 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:51.519042 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:51.519044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:51.519480 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:51.519753 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:51.520087 (+ 334us) server_negotiation.cc:299] Negotiation successful
0806 02:52:51.520297 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 02:52:51.521431 4984 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:52:51.644449 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 4801
I20260806 02:52:51.674486 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 3809
I20260806 02:52:51.675693 4063 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:52:51.794318 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 3809
I20260806 02:52:51.821105 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 3961
I20260806 02:52:51.821342 5191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:51.820564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45807 (local address 127.0.0.1:41730)
0806 02:52:51.821061 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:51.821155 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":264,"threads_started":1}
I20260806 02:52:51.822299 4362 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:52:51.822698 5191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:51.822249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45807 (local address 127.0.0.1:41732)
0806 02:52:51.822415 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:51.822513 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260806 02:52:51.946041 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 3961
2026-08-06T02:52:51Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (10566 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:52:52 dist-test-slave-zj8q krb5kdc[5196](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:52:52 dist-test-slave-zj8q krb5kdc[5196](info): set up 2 sockets
Aug 06 02:52:52 dist-test-slave-zj8q krb5kdc[5196](info): commencing operation
krb5kdc: starting...
W20260806 02:52:53.188441 26446 mini_kdc.cc:121] Time spent starting KDC: real 1.191s user 0.000s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Aug 06 02:52:53 dist-test-slave-zj8q krb5kdc[5196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:52:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:52:53Z Disabled control of system clock
WARNING: no policy specified for oryx/127.25.211.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.211.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.211.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/krb5kdc/oryx_127.25.211.190.keytab.
Entry for principal oryx/127.25.211.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/krb5kdc/oryx_127.25.211.190.keytab.
I20260806 02:52:53.478061 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:37245
--webserver_interface=127.25.211.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/krb5kdc/oryx_127.25.211.190.keytab
--principal=oryx/127.25.211.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:41041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.211.190:37245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:52:53.832804 5216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:52:53.833135 5216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:52:53.833230 5216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:52:53.840916 5216 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:52:53.841051 5216 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 02:52:53.841094 5216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:52:53.841157 5216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:52:53.841208 5216 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:52:53.841280 5216 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:52:53.851753 5216 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:41041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.211.190:37245
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/krb5kdc/oryx_127.25.211.190.keytab
--principal=oryx/127.25.211.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:37245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.211.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:52:53.853085 5216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:52:53.854466 5216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:52:53.866995 5222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:53.866904 5223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:53.866902 5225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:52:53.868155 5216 server_base.cc:1034] running on GCE node
I20260806 02:52:53.869042 5216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:52:53.870664 5216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:52:53.872038 5216 hybrid_clock.cc:648] HybridClock initialized: now 1785984773871958 us; error 49 us; skew 500 ppm
I20260806 02:52:53.872404 5216 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:52:53 dist-test-slave-zj8q krb5kdc[5196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984773, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.211.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:52:53.875867 5216 init.cc:377] Logged in from keytab as oryx/127.25.211.190@KRBTEST.COM (short username oryx)
I20260806 02:52:53.877398 5216 webserver.cc:460] Webserver started at http://127.25.211.190:33539/ using document root <none> and password file <none>
I20260806 02:52:53.878121 5216 fs_manager.cc:359] Metadata directory not provided
I20260806 02:52:53.878273 5216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:52:53.878618 5216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:52:53.881767 5216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/master-0/data/instance:
uuid: "8c91b83144bf42b38480be74a574bb17"
format_stamp: "Formatted at 2026-08-06 02:52:53 on dist-test-slave-zj8q"
server_key: "f32f5a854493e960cab30bb2a9ae99fb"
server_key_iv: "f709f69dfe5d97428eee02a30e388161"
server_key_version: "encryptionkey@0"
I20260806 02:52:53.882584 5216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/master-0/wal/instance:
uuid: "8c91b83144bf42b38480be74a574bb17"
format_stamp: "Formatted at 2026-08-06 02:52:53 on dist-test-slave-zj8q"
server_key: "f32f5a854493e960cab30bb2a9ae99fb"
server_key_iv: "f709f69dfe5d97428eee02a30e388161"
server_key_version: "encryptionkey@0"
I20260806 02:52:53.888182 5216 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 02:52:53.892366 5233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:53.893801 5216 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:52:53.893985 5216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/master-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/master-0/wal
uuid: "8c91b83144bf42b38480be74a574bb17"
format_stamp: "Formatted at 2026-08-06 02:52:53 on dist-test-slave-zj8q"
server_key: "f32f5a854493e960cab30bb2a9ae99fb"
server_key_iv: "f709f69dfe5d97428eee02a30e388161"
server_key_version: "encryptionkey@0"
I20260806 02:52:53.894248 5216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:52:53.924867 5216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:52:53.929625 5216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:52:53.929917 5216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:52:53.950825 5216 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.190:37245
I20260806 02:52:53.950848 5321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.190:37245 every 8 connection(s)
I20260806 02:52:53.952315 5216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
I20260806 02:52:53.956671 5322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:53.958145 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 5216
I20260806 02:52:53.958416 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/master-0/wal/instance
I20260806 02:52:53.959270 26446 external_mini_cluster.cc:1428] Setting key d90570af6eb9c34ae09921988384b3d1
I20260806 02:52:53.970914 5322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c91b83144bf42b38480be74a574bb17: Bootstrap starting.
Aug 06 02:52:53 dist-test-slave-zj8q krb5kdc[5196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM
I20260806 02:52:53.974581 5322 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c91b83144bf42b38480be74a574bb17: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:53.975888 5322 log.cc:824] T 00000000000000000000000000000000 P 8c91b83144bf42b38480be74a574bb17: Log is configured to *not* fsync() on all Append() calls
I20260806 02:52:53.979584 5322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c91b83144bf42b38480be74a574bb17: No bootstrap required, opened a new log
I20260806 02:52:53.983037 5327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:53.961560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56606 (local address 127.25.211.190:37245)
0806 02:52:53.962405 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:53.962420 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:53.962466 (+ 46us) server_negotiation.cc:407] Connection header received
0806 02:52:53.964112 (+ 1646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:53.964141 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:53.964597 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:53.965005 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:53.966646 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:53.968037 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:53.969365 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:53.970124 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:53.970333 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:53.974641 (+ 4308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:53.974651 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:53.974673 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:53.974746 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:53.978339 (+ 3593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:53.979363 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:53.979369 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:53.979376 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:53.979583 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:53.980057 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:53.980060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:53.980062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:53.980651 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:53.980891 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:53.981389 (+ 498us) server_negotiation.cc:299] Negotiation successful
0806 02:52:53.981802 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":345,"threads_started":1}
I20260806 02:52:53.989301 5322 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c91b83144bf42b38480be74a574bb17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c91b83144bf42b38480be74a574bb17" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 37245 } }
I20260806 02:52:53.989775 5322 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c91b83144bf42b38480be74a574bb17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:53.989913 5322 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c91b83144bf42b38480be74a574bb17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c91b83144bf42b38480be74a574bb17, State: Initialized, Role: FOLLOWER
I20260806 02:52:53.990566 5322 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c91b83144bf42b38480be74a574bb17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c91b83144bf42b38480be74a574bb17" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 37245 } }
I20260806 02:52:53.990825 5322 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c91b83144bf42b38480be74a574bb17 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:52:53.990962 5322 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c91b83144bf42b38480be74a574bb17 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:52:53.991129 5322 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c91b83144bf42b38480be74a574bb17 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:53.993410 5322 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c91b83144bf42b38480be74a574bb17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c91b83144bf42b38480be74a574bb17" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 37245 } }
I20260806 02:52:53.994015 5322 leader_election.cc:302] T 00000000000000000000000000000000 P 8c91b83144bf42b38480be74a574bb17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c91b83144bf42b38480be74a574bb17; no voters:
I20260806 02:52:53.994701 5322 leader_election.cc:288] T 00000000000000000000000000000000 P 8c91b83144bf42b38480be74a574bb17 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:52:53.994851 5330 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c91b83144bf42b38480be74a574bb17 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:52:53.995585 5330 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c91b83144bf42b38480be74a574bb17 [term 1 LEADER]: Becoming Leader. State: Replica: 8c91b83144bf42b38480be74a574bb17, State: Running, Role: LEADER
I20260806 02:52:53.996284 5330 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c91b83144bf42b38480be74a574bb17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c91b83144bf42b38480be74a574bb17" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 37245 } }
I20260806 02:52:53.997094 5322 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c91b83144bf42b38480be74a574bb17 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:52:53.999336 5331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c91b83144bf42b38480be74a574bb17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c91b83144bf42b38480be74a574bb17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c91b83144bf42b38480be74a574bb17" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 37245 } } }
I20260806 02:52:53.999755 5331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c91b83144bf42b38480be74a574bb17 [sys.catalog]: This master's current role is: LEADER
I20260806 02:52:54.000401 5332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c91b83144bf42b38480be74a574bb17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c91b83144bf42b38480be74a574bb17. Latest consensus state: current_term: 1 leader_uuid: "8c91b83144bf42b38480be74a574bb17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c91b83144bf42b38480be74a574bb17" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 37245 } } }
I20260806 02:52:54.000713 5332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c91b83144bf42b38480be74a574bb17 [sys.catalog]: This master's current role is: LEADER
I20260806 02:52:54.000975 5339 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:52:54.007493 5339 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:52:54.020638 5339 catalog_manager.cc:1347] Generated new cluster ID: afa72dc25db94037aa1f6cf6d6f36407
I20260806 02:52:54.020768 5339 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:52:54.031517 5339 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:52:54.033581 5339 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:52:54.045600 5339 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c91b83144bf42b38480be74a574bb17: Generated new TSK 0
I20260806 02:52:54.046888 5339 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.25.211.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.211.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.211.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/krb5kdc/oryx_127.25.211.129.keytab.
Entry for principal oryx/127.25.211.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/krb5kdc/oryx_127.25.211.129.keytab.
I20260806 02:52:54.113476 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.129:0
--local_ip_for_outbound_sockets=127.25.211.129
--webserver_interface=127.25.211.129
--webserver_port=0
--tserver_master_addrs=127.25.211.190:37245
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/krb5kdc/oryx_127.25.211.129.keytab
--principal=oryx/127.25.211.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:41041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:52:54.479648 5357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:52:54.479928 5357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:52:54.480026 5357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:52:54.487461 5357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:52:54.487591 5357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:52:54.487687 5357 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 02:52:54.487769 5357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.129
I20260806 02:52:54.498087 5357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:41041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/krb5kdc/oryx_127.25.211.129.keytab
--principal=oryx/127.25.211.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.211.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.25.211.190:37245
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.129
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:52:54.499656 5357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:52:54.501067 5357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:52:54.514735 5364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:54.514843 5365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:54.515399 5367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:52:54.516564 5357 server_base.cc:1034] running on GCE node
I20260806 02:52:54.517267 5357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:52:54.518311 5357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:52:54.519539 5357 hybrid_clock.cc:648] HybridClock initialized: now 1785984774519476 us; error 43 us; skew 500 ppm
I20260806 02:52:54.519865 5357 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:52:54 dist-test-slave-zj8q krb5kdc[5196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984774, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.211.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:52:54.527841 5357 init.cc:377] Logged in from keytab as oryx/127.25.211.129@KRBTEST.COM (short username oryx)
I20260806 02:52:54.529452 5357 webserver.cc:460] Webserver started at http://127.25.211.129:37595/ using document root <none> and password file <none>
I20260806 02:52:54.530306 5357 fs_manager.cc:359] Metadata directory not provided
I20260806 02:52:54.530472 5357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:52:54.530833 5357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:52:54.533998 5357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/ts-0/data/instance:
uuid: "ab9b2267b3dc4fc794f9a3f59345fc80"
format_stamp: "Formatted at 2026-08-06 02:52:54 on dist-test-slave-zj8q"
server_key: "25ce67f7516a85561662474bdc48c67a"
server_key_iv: "b87da8348d3cd950f599cefb901a1d0a"
server_key_version: "encryptionkey@0"
I20260806 02:52:54.534905 5357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/ts-0/wal/instance:
uuid: "ab9b2267b3dc4fc794f9a3f59345fc80"
format_stamp: "Formatted at 2026-08-06 02:52:54 on dist-test-slave-zj8q"
server_key: "25ce67f7516a85561662474bdc48c67a"
server_key_iv: "b87da8348d3cd950f599cefb901a1d0a"
server_key_version: "encryptionkey@0"
I20260806 02:52:54.540483 5357 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 02:52:54.544559 5379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:54.546125 5357 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:52:54.546391 5357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/ts-0/wal
uuid: "ab9b2267b3dc4fc794f9a3f59345fc80"
format_stamp: "Formatted at 2026-08-06 02:52:54 on dist-test-slave-zj8q"
server_key: "25ce67f7516a85561662474bdc48c67a"
server_key_iv: "b87da8348d3cd950f599cefb901a1d0a"
server_key_version: "encryptionkey@0"
I20260806 02:52:54.546623 5357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:52:54.560911 5357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:52:54.565682 5357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:52:54.565971 5357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:52:54.568915 5357 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:52:54.569015 5357 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:54.569146 5357 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:52:54.569222 5357 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 02:52:54 dist-test-slave-zj8q krb5kdc[5196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984774, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.211.129@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM
I20260806 02:52:54.594919 5394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:54.569941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:41963 (local address 127.25.211.190:37245)
0806 02:52:54.570479 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:54.570487 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:54.571776 (+ 1289us) server_negotiation.cc:407] Connection header received
0806 02:52:54.575100 (+ 3324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:54.575106 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:54.575213 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:54.575348 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:54.578265 (+ 2917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:54.579269 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:54.582587 (+ 3318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:54.583124 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:54.583285 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:54.588306 (+ 5021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:54.588315 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:54.588321 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:54.588363 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:54.591510 (+ 3147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:54.592354 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:54.592358 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:54.592361 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:54.592483 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:54.592987 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:54.592990 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:54.592992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:54.593274 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:54.593412 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:54.594298 (+ 886us) server_negotiation.cc:299] Negotiation successful
0806 02:52:54.594537 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":278,"threads_started":1}
I20260806 02:52:54.596485 5397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:54.570626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:37245 (local address 127.25.211.129:41963)
0806 02:52:54.571556 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:54.571622 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:54.574609 (+ 2987us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:54.575718 (+ 1109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:54.575734 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:54.576796 (+ 1062us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:54.577971 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:54.578005 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:54.579422 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:54.579428 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:54.582359 (+ 2931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:54.582410 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:54.583426 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:54.583435 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:54.583803 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:54.584739 (+ 936us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:52:54.584763 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:54.587927 (+ 3164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:54.591697 (+ 3770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:54.591708 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:54.591730 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:54.592214 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:54.592634 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:54.592640 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:54.592644 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:54.592868 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:54.593413 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:54.593421 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:54.593858 (+ 437us) client_negotiation.cc:769] Sending connection context
0806 02:52:54.594119 (+ 261us) client_negotiation.cc:240] Negotiation successful
0806 02:52:54.594799 (+ 680us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":353,"threads_started":1}
I20260806 02:52:54.615372 5357 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.129:42013
I20260806 02:52:54.615396 5557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.129:42013 every 8 connection(s)
I20260806 02:52:54.616797 5357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
I20260806 02:52:54.618057 5394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:54.603796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:51887 (local address 127.25.211.190:37245)
0806 02:52:54.604120 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:54.604127 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:54.604932 (+ 805us) server_negotiation.cc:407] Connection header received
0806 02:52:54.605126 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:54.605133 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:54.605233 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:54.605350 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:54.606419 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:54.607283 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:54.609019 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:54.609606 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:54.609740 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:54.612383 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:54.612388 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:54.612391 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:54.612427 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:54.615274 (+ 2847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:54.616007 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:54.616012 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:54.616014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:54.616155 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:54.616511 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:54.616516 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:54.616519 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:54.616816 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:54.616949 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:54.617444 (+ 495us) server_negotiation.cc:299] Negotiation successful
0806 02:52:54.617688 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195}
I20260806 02:52:54.618206 5520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:54.604136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:37245 (local address 127.25.211.129:51887)
0806 02:52:54.604793 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:54.604814 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:54.604991 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:54.605363 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:54.605367 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:54.605880 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:54.606254 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:54.606268 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:54.607444 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:54.607450 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:54.608868 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:54.608890 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:54.610000 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:54.610005 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:54.610185 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:54.610998 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:52:54.611028 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:54.612106 (+ 1078us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:54.615411 (+ 3305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:54.615417 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:54.615422 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:54.615889 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:54.616268 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:54.616273 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:54.616277 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:54.616409 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:54.617004 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:54.617008 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:54.617186 (+ 178us) client_negotiation.cc:769] Sending connection context
0806 02:52:54.617318 (+ 132us) client_negotiation.cc:240] Negotiation successful
0806 02:52:54.617698 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":318,"threads_started":1}
I20260806 02:52:54.619338 5559 heartbeater.cc:347] Connected to a master server at 127.25.211.190:37245
I20260806 02:52:54.619638 5559 heartbeater.cc:464] Registering TS with master...
I20260806 02:52:54.620668 5559 heartbeater.cc:511] Master 127.25.211.190:37245 requested a full tablet report, sending...
I20260806 02:52:54.623169 5255 ts_manager.cc:194] Registered new tserver with Master: ab9b2267b3dc4fc794f9a3f59345fc80 (127.25.211.129:42013)
I20260806 02:52:54.624752 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 5357
I20260806 02:52:54.624786 5255 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.25.211.129@KRBTEST.COM'} at 127.25.211.129:41963
I20260806 02:52:54.624931 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/ts-0/wal/instance
I20260806 02:52:54.625648 26446 external_mini_cluster.cc:1428] Setting key 0fe44ddd7b40af7c3c486d61f662ec50
WARNING: no policy specified for oryx/127.25.211.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.211.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.211.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/krb5kdc/oryx_127.25.211.130.keytab.
Entry for principal oryx/127.25.211.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/krb5kdc/oryx_127.25.211.130.keytab.
I20260806 02:52:54.686043 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.130:0
--local_ip_for_outbound_sockets=127.25.211.130
--webserver_interface=127.25.211.130
--webserver_port=0
--tserver_master_addrs=127.25.211.190:37245
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/krb5kdc/oryx_127.25.211.130.keytab
--principal=oryx/127.25.211.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:41041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:52:55.040328 5563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:52:55.040683 5563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:52:55.040829 5563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:52:55.048442 5563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:52:55.048605 5563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:52:55.048735 5563 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 02:52:55.048859 5563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.130
I20260806 02:52:55.059355 5563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:41041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/krb5kdc/oryx_127.25.211.130.keytab
--principal=oryx/127.25.211.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.211.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.25.211.190:37245
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.130
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:52:55.060838 5563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:52:55.062326 5563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:52:55.076293 5580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:55.076342 5576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:55.076619 5574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:52:55.077551 5563 server_base.cc:1034] running on GCE node
I20260806 02:52:55.078301 5563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:52:55.079409 5563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:52:55.080657 5563 hybrid_clock.cc:648] HybridClock initialized: now 1785984775080579 us; error 49 us; skew 500 ppm
I20260806 02:52:55.080964 5563 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:52:55 dist-test-slave-zj8q krb5kdc[5196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984775, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.211.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:52:55.084306 5563 init.cc:377] Logged in from keytab as oryx/127.25.211.130@KRBTEST.COM (short username oryx)
I20260806 02:52:55.085835 5563 webserver.cc:460] Webserver started at http://127.25.211.130:46435/ using document root <none> and password file <none>
I20260806 02:52:55.086597 5563 fs_manager.cc:359] Metadata directory not provided
I20260806 02:52:55.086694 5563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:52:55.086983 5563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:52:55.090016 5563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/ts-1/data/instance:
uuid: "90c9940f7ea24833bb84133bf92d11df"
format_stamp: "Formatted at 2026-08-06 02:52:55 on dist-test-slave-zj8q"
server_key: "780d1cb3b8e5a58e5af7e016eadb4f08"
server_key_iv: "78bb7d62b0a08b5cb9587a7298ea0b0a"
server_key_version: "encryptionkey@0"
I20260806 02:52:55.090940 5563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/ts-1/wal/instance:
uuid: "90c9940f7ea24833bb84133bf92d11df"
format_stamp: "Formatted at 2026-08-06 02:52:55 on dist-test-slave-zj8q"
server_key: "780d1cb3b8e5a58e5af7e016eadb4f08"
server_key_iv: "78bb7d62b0a08b5cb9587a7298ea0b0a"
server_key_version: "encryptionkey@0"
I20260806 02:52:55.096738 5563 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 02:52:55.101152 5593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:55.102721 5563 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:52:55.102911 5563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/ts-1/wal
uuid: "90c9940f7ea24833bb84133bf92d11df"
format_stamp: "Formatted at 2026-08-06 02:52:55 on dist-test-slave-zj8q"
server_key: "780d1cb3b8e5a58e5af7e016eadb4f08"
server_key_iv: "78bb7d62b0a08b5cb9587a7298ea0b0a"
server_key_version: "encryptionkey@0"
I20260806 02:52:55.103156 5563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:52:55.131567 5563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:52:55.136448 5563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:52:55.136772 5563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:52:55.139983 5563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:52:55.140290 5563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:55.140468 5563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:52:55.140552 5563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 02:52:55 dist-test-slave-zj8q krb5kdc[5196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984775, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.211.130@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM
I20260806 02:52:55.166561 5601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:55.140560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:43431 (local address 127.25.211.190:37245)
0806 02:52:55.141086 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:55.141093 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:55.142405 (+ 1312us) server_negotiation.cc:407] Connection header received
0806 02:52:55.144913 (+ 2508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:55.144919 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:55.145025 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:55.145163 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:55.147823 (+ 2660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:55.149073 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:55.152007 (+ 2934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:55.152915 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:55.153087 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:55.158787 (+ 5700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:55.158796 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:55.158802 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:55.158858 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:55.162732 (+ 3874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:55.163667 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:55.163675 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:55.163679 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:55.163878 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:55.164373 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:55.164378 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:55.164382 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:55.164810 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:55.164990 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:55.165806 (+ 816us) server_negotiation.cc:299] Negotiation successful
0806 02:52:55.166106 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":238,"threads_started":1}
I20260806 02:52:55.167832 5604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:55.141228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:37245 (local address 127.25.211.130:43431)
0806 02:52:55.142193 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:55.142254 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:55.144501 (+ 2247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:55.145635 (+ 1134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:55.145644 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:55.146545 (+ 901us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:55.147488 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:55.147510 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:55.149304 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:55.149309 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:55.151738 (+ 2429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:55.151764 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:55.153274 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:55.153287 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:55.153685 (+ 398us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:55.154626 (+ 941us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:52:55.154668 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:55.158367 (+ 3699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:55.162955 (+ 4588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:55.162965 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:55.162981 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:55.163503 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:55.163994 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:55.163998 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:55.164001 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:55.164252 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:55.164940 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:55.164947 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:55.165361 (+ 414us) client_negotiation.cc:769] Sending connection context
0806 02:52:55.165624 (+ 263us) client_negotiation.cc:240] Negotiation successful
0806 02:52:55.166075 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":382,"threads_started":1}
I20260806 02:52:55.188676 5563 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.130:40117
I20260806 02:52:55.188695 5773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.130:40117 every 8 connection(s)
I20260806 02:52:55.190086 5563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
I20260806 02:52:55.190443 5601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:55.174606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:37891 (local address 127.25.211.190:37245)
0806 02:52:55.174780 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:55.174785 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:55.175484 (+ 699us) server_negotiation.cc:407] Connection header received
0806 02:52:55.175674 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:55.175680 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:55.175768 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:55.175880 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:55.177277 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:55.178221 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:55.180368 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:55.180898 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:55.181046 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:55.184068 (+ 3022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:55.184099 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:55.184104 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:55.184152 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:55.187438 (+ 3286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:55.188316 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:55.188321 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:55.188323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:55.188450 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:55.188859 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:55.188862 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:55.188864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:55.189236 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:55.189389 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:55.189808 (+ 419us) server_negotiation.cc:299] Negotiation successful
0806 02:52:55.190035 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 02:52:55.190688 5733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:55.174738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:37245 (local address 127.25.211.130:37891)
0806 02:52:55.175361 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:55.175379 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:55.175536 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:55.175959 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:55.175965 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:55.176643 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:55.177092 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:55.177108 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:55.178427 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:55.178432 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:55.180163 (+ 1731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:55.180186 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:55.181144 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:55.181152 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:55.181368 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:55.182114 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:52:55.182141 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:55.183785 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:55.187600 (+ 3815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:55.187606 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:55.187611 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:55.188179 (+ 568us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:55.188577 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:55.188582 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:55.188586 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:55.188741 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:55.189362 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:55.189366 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:55.189555 (+ 189us) client_negotiation.cc:769] Sending connection context
0806 02:52:55.189679 (+ 124us) client_negotiation.cc:240] Negotiation successful
0806 02:52:55.190049 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":333,"threads_started":1}
I20260806 02:52:55.193735 5774 heartbeater.cc:347] Connected to a master server at 127.25.211.190:37245
I20260806 02:52:55.193969 5774 heartbeater.cc:464] Registering TS with master...
I20260806 02:52:55.194698 5774 heartbeater.cc:511] Master 127.25.211.190:37245 requested a full tablet report, sending...
I20260806 02:52:55.195336 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 5563
I20260806 02:52:55.195478 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/ts-1/wal/instance
I20260806 02:52:55.195959 26446 external_mini_cluster.cc:1428] Setting key 5227369992cf8fa470ddca3cc0f16522
I20260806 02:52:55.196444 5255 ts_manager.cc:194] Registered new tserver with Master: 90c9940f7ea24833bb84133bf92d11df (127.25.211.130:40117)
I20260806 02:52:55.197593 5255 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.25.211.130@KRBTEST.COM'} at 127.25.211.130:43431
WARNING: no policy specified for oryx/127.25.211.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.211.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.211.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/krb5kdc/oryx_127.25.211.131.keytab.
Entry for principal oryx/127.25.211.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/krb5kdc/oryx_127.25.211.131.keytab.
I20260806 02:52:55.259626 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.131:0
--local_ip_for_outbound_sockets=127.25.211.131
--webserver_interface=127.25.211.131
--webserver_port=0
--tserver_master_addrs=127.25.211.190:37245
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/krb5kdc/oryx_127.25.211.131.keytab
--principal=oryx/127.25.211.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:41041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:52:55.614486 5780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:52:55.614902 5780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:52:55.615242 5780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:52:55.623229 5780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:52:55.623377 5780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:52:55.623494 5780 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 02:52:55.623610 5780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.131
I20260806 02:52:55.627650 5559 heartbeater.cc:503] Master 127.25.211.190:37245 was elected leader, sending a full tablet report...
I20260806 02:52:55.634217 5780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:41041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/krb5kdc/oryx_127.25.211.131.keytab
--principal=oryx/127.25.211.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.211.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.25.211.190:37245
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.131
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:52:55.635658 5780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:52:55.637090 5780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:52:55.651201 5786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:55.651094 5790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:52:55.651243 5794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:52:55.652158 5780 server_base.cc:1034] running on GCE node
I20260806 02:52:55.652858 5780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:52:55.653878 5780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:52:55.655119 5780 hybrid_clock.cc:648] HybridClock initialized: now 1785984775655028 us; error 66 us; skew 500 ppm
I20260806 02:52:55.655443 5780 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:52:55 dist-test-slave-zj8q krb5kdc[5196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984775, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.211.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:52:55.658694 5780 init.cc:377] Logged in from keytab as oryx/127.25.211.131@KRBTEST.COM (short username oryx)
I20260806 02:52:55.660313 5780 webserver.cc:460] Webserver started at http://127.25.211.131:36885/ using document root <none> and password file <none>
I20260806 02:52:55.661070 5780 fs_manager.cc:359] Metadata directory not provided
I20260806 02:52:55.661180 5780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:52:55.661520 5780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:52:55.664781 5780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/ts-2/data/instance:
uuid: "048e0815a5df436985507ef1fae29d80"
format_stamp: "Formatted at 2026-08-06 02:52:55 on dist-test-slave-zj8q"
server_key: "184ec6de4b5ce01865f38b6f9137033e"
server_key_iv: "79a9004efd29e57c68af90d0cf324466"
server_key_version: "encryptionkey@0"
I20260806 02:52:55.665829 5780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/ts-2/wal/instance:
uuid: "048e0815a5df436985507ef1fae29d80"
format_stamp: "Formatted at 2026-08-06 02:52:55 on dist-test-slave-zj8q"
server_key: "184ec6de4b5ce01865f38b6f9137033e"
server_key_iv: "79a9004efd29e57c68af90d0cf324466"
server_key_version: "encryptionkey@0"
I20260806 02:52:55.671730 5780 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260806 02:52:55.675848 5803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:55.677444 5780 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260806 02:52:55.677655 5780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/ts-2/wal
uuid: "048e0815a5df436985507ef1fae29d80"
format_stamp: "Formatted at 2026-08-06 02:52:55 on dist-test-slave-zj8q"
server_key: "184ec6de4b5ce01865f38b6f9137033e"
server_key_iv: "79a9004efd29e57c68af90d0cf324466"
server_key_version: "encryptionkey@0"
I20260806 02:52:55.677908 5780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:52:55.711550 5780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:52:55.715931 5780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:52:55.716218 5780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:52:55.719184 5780 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:52:55.719296 5780 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:52:55.719434 5780 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:52:55.719502 5780 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 02:52:55 dist-test-slave-zj8q krb5kdc[5196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984775, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.211.131@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM
I20260806 02:52:55.745395 5823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:55.719884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:37571 (local address 127.25.211.190:37245)
0806 02:52:55.720501 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:55.720506 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:55.721641 (+ 1135us) server_negotiation.cc:407] Connection header received
0806 02:52:55.723804 (+ 2163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:55.723810 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:55.723899 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:55.723998 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:55.726571 (+ 2573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:55.727607 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:55.731084 (+ 3477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:55.731682 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:55.731835 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:55.737181 (+ 5346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:55.737189 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:55.737194 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:55.737249 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:55.740982 (+ 3733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:55.742182 (+ 1200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:55.742189 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:55.742193 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:55.742355 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:55.743002 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:55.743006 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:55.743008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:55.743397 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:55.743552 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:55.744603 (+ 1051us) server_negotiation.cc:299] Negotiation successful
0806 02:52:55.744900 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":395,"threads_started":1}
I20260806 02:52:55.746295 5826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:55.720586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:37245 (local address 127.25.211.131:37571)
0806 02:52:55.721446 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:55.721487 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:55.723464 (+ 1977us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:55.724369 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:55.724385 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:55.725265 (+ 880us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:55.726335 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:55.726365 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:55.727825 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:55.727832 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:55.730813 (+ 2981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:55.730840 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:55.732075 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:55.732086 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:55.732445 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:55.733362 (+ 917us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:52:55.733387 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:55.736787 (+ 3400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:55.741225 (+ 4438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:55.741237 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:55.741264 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:55.741998 (+ 734us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:55.742487 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:55.742490 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:55.742493 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:55.742849 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:55.743522 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:55.743531 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:55.744101 (+ 570us) client_negotiation.cc:769] Sending connection context
0806 02:52:55.744446 (+ 345us) client_negotiation.cc:240] Negotiation successful
0806 02:52:55.744961 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":351,"threads_started":1}
I20260806 02:52:55.769229 5954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:55.753842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:37245 (local address 127.25.211.131:44307)
0806 02:52:55.754521 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:55.754541 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:55.754726 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:55.755187 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:55.755191 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:55.755676 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:55.755977 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:55.755987 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:55.757551 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:55.757555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:55.758992 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:55.759013 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:55.760101 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:55.760108 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:55.760325 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:55.761012 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:52:55.761034 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:55.762179 (+ 1145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:55.765953 (+ 3774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:55.765959 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:55.765966 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:55.766597 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:55.767036 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:55.767041 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:55.767049 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:55.767234 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:55.768044 (+ 810us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:55.768048 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:55.768283 (+ 235us) client_negotiation.cc:769] Sending connection context
0806 02:52:55.768466 (+ 183us) client_negotiation.cc:240] Negotiation successful
0806 02:52:55.768835 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":383,"threads_started":1}
I20260806 02:52:55.769313 5823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:55.753494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:44307 (local address 127.25.211.190:37245)
0806 02:52:55.753713 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:55.753720 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:55.754662 (+ 942us) server_negotiation.cc:407] Connection header received
0806 02:52:55.754890 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:55.754897 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:55.754992 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:55.755133 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:55.756148 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:55.757369 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:55.759195 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:55.759878 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:55.760038 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:55.762441 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:55.762448 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:55.762453 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:55.762496 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:55.765789 (+ 3293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:55.766756 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:55.766761 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:55.766764 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:55.766905 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:55.767402 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:55.767409 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:55.767412 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:55.767807 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:55.767980 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:55.768662 (+ 682us) server_negotiation.cc:299] Negotiation successful
0806 02:52:55.768939 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 02:52:55.771090 5780 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.131:38389
I20260806 02:52:55.771251 5999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.131:38389 every 8 connection(s)
I20260806 02:52:55.772478 5780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
I20260806 02:52:55.774847 6001 heartbeater.cc:347] Connected to a master server at 127.25.211.190:37245
I20260806 02:52:55.775094 6001 heartbeater.cc:464] Registering TS with master...
I20260806 02:52:55.775755 6001 heartbeater.cc:511] Master 127.25.211.190:37245 requested a full tablet report, sending...
I20260806 02:52:55.777129 5255 ts_manager.cc:194] Registered new tserver with Master: 048e0815a5df436985507ef1fae29d80 (127.25.211.131:38389)
I20260806 02:52:55.777966 5255 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.25.211.131@KRBTEST.COM'} at 127.25.211.131:37571
I20260806 02:52:55.779449 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 5780
I20260806 02:52:55.779652 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/ts-2/wal/instance
I20260806 02:52:55.780210 26446 external_mini_cluster.cc:1428] Setting key 3264ecf46176ca324fd9a145bb1d2914
I20260806 02:52:55.783378 26446 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 02:52:55 dist-test-slave-zj8q krb5kdc[5196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984775, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 02:52:55 dist-test-slave-zj8q krb5kdc[5196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:52:55 dist-test-slave-zj8q krb5kdc[5196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
W20260806 02:52:55.845181 5823 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:52:55.833685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56608 (local address 127.25.211.190:37245)
0806 02:52:55.833890 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:55.833895 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:55.834124 (+ 229us) server_negotiation.cc:407] Connection header received
0806 02:52:55.834331 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:55.834335 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:55.834421 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:55.834517 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:55.835991 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:55.836781 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:55.838061 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:55.838553 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:55.838659 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:55.844880 (+ 6221us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56608: BlockingRecv error: recv got EOF from 127.0.0.1:56608 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Aug 06 02:52:55 dist-test-slave-zj8q krb5kdc[5196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984775, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM
I20260806 02:52:55.863184 5823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:55.848396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56614 (local address 127.25.211.190:37245)
0806 02:52:55.848658 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:55.848665 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:55.848873 (+ 208us) server_negotiation.cc:407] Connection header received
0806 02:52:55.849077 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:55.849082 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:55.849153 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:55.849239 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:55.850706 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:55.851472 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:55.852861 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:55.853298 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:55.853419 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:55.857638 (+ 4219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:55.857644 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:55.857648 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:55.857694 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:55.860370 (+ 2676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:55.861189 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:55.861195 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:55.861199 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:55.861349 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:55.861822 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:55.861827 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:55.861829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:55.862177 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:55.862413 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:55.862690 (+ 277us) server_negotiation.cc:299] Negotiation successful
0806 02:52:55.862900 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260806 02:52:55.867594 5255 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56614:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:52:55.874836 5255 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:52:55.899211 6033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:55.888914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.131:38389 (local address 127.0.0.1:37790)
0806 02:52:55.890048 (+ 1134us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:55.890089 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:55.890339 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:55.891459 (+ 1120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:55.891468 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:55.891496 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:52:55.892244 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:55.892261 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:55.894135 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:55.894140 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:55.896266 (+ 2126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:55.896278 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:55.897917 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:55.897924 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:55.898126 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:55.898233 (+ 107us) client_negotiation.cc:769] Sending connection context
0806 02:52:55.898480 (+ 247us) client_negotiation.cc:240] Negotiation successful
0806 02:52:55.898801 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1010,"thread_start_us":692,"threads_started":1}
I20260806 02:52:55.899199 6030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:55.888575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59784 (local address 127.25.211.130:40117)
0806 02:52:55.889112 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:55.889122 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:55.890244 (+ 1122us) server_negotiation.cc:407] Connection header received
0806 02:52:55.890633 (+ 389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:55.890641 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:55.890873 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:55.891060 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:52:55.891927 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:55.893399 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:55.896335 (+ 2936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:55.897918 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:55.898081 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:55.898410 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:55.898712 (+ 302us) server_negotiation.cc:299] Negotiation successful
0806 02:52:55.898899 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":264,"threads_started":1}
I20260806 02:52:55.900025 6034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:55.888142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.130:40117 (local address 127.0.0.1:59784)
0806 02:52:55.890047 (+ 1905us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:55.890089 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:55.890371 (+ 282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:55.891128 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:55.891136 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:55.891177 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:52:55.891676 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:55.891690 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:55.893574 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:55.893579 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:55.896113 (+ 2534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:55.896131 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:55.898224 (+ 2093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:55.898231 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:55.898468 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:55.898500 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 02:52:55.898573 (+ 73us) client_negotiation.cc:240] Negotiation successful
0806 02:52:55.899618 (+ 1045us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1616,"mutex_wait_us":888,"thread_start_us":2122,"threads_started":1}
I20260806 02:52:55.900684 6031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:55.888694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37790 (local address 127.25.211.131:38389)
0806 02:52:55.889608 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:55.889616 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:55.890244 (+ 628us) server_negotiation.cc:407] Connection header received
0806 02:52:55.890566 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:55.890572 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:55.890782 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:55.890949 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:52:55.892451 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:55.893966 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:55.896439 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:55.897615 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:55.897808 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:55.898024 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:55.900127 (+ 2103us) server_negotiation.cc:299] Negotiation successful
0806 02:52:55.900341 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":737,"thread_start_us":571,"threads_started":1}
I20260806 02:52:55.901088 6035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:55.891166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:42013 (local address 127.0.0.1:34414)
0806 02:52:55.892059 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:55.892087 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:55.892212 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:55.893396 (+ 1184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:55.893401 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:55.893429 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:52:55.893998 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:55.894012 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:55.895710 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:55.895715 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:55.897884 (+ 2169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:55.897904 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:55.900334 (+ 2430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:55.900341 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:55.900528 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:55.900561 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 02:52:55.900667 (+ 106us) client_negotiation.cc:240] Negotiation successful
0806 02:52:55.900811 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":629,"threads_started":1}
I20260806 02:52:55.901868 6037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:55.891627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34414 (local address 127.25.211.129:42013)
0806 02:52:55.892768 (+ 1141us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:55.892776 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:55.892806 (+ 30us) server_negotiation.cc:407] Connection header received
0806 02:52:55.892918 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:55.892926 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:55.893171 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:55.893372 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:52:55.894235 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:55.895538 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:55.898104 (+ 2566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:55.899778 (+ 1674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:55.899984 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:55.900166 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:55.901391 (+ 1225us) server_negotiation.cc:299] Negotiation successful
0806 02:52:55.901590 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":967,"thread_start_us":891,"threads_started":1}
I20260806 02:52:55.904414 5893 tablet_service.cc:1467] Processing CreateTablet for tablet 4effcda7f378455694b9781859a2f7d7 (DEFAULT_TABLE table=test-table [id=bd81223bc03b450095387fe90f4c0121]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:52:55.904495 5672 tablet_service.cc:1467] Processing CreateTablet for tablet 4effcda7f378455694b9781859a2f7d7 (DEFAULT_TABLE table=test-table [id=bd81223bc03b450095387fe90f4c0121]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:52:55.905992 5672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4effcda7f378455694b9781859a2f7d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:55.906059 5893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4effcda7f378455694b9781859a2f7d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:55.905625 5458 tablet_service.cc:1467] Processing CreateTablet for tablet 4effcda7f378455694b9781859a2f7d7 (DEFAULT_TABLE table=test-table [id=bd81223bc03b450095387fe90f4c0121]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:52:55.907398 5458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4effcda7f378455694b9781859a2f7d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:55.919035 6039 tablet_bootstrap.cc:492] T 4effcda7f378455694b9781859a2f7d7 P 90c9940f7ea24833bb84133bf92d11df: Bootstrap starting.
I20260806 02:52:55.921113 6040 tablet_bootstrap.cc:492] T 4effcda7f378455694b9781859a2f7d7 P 048e0815a5df436985507ef1fae29d80: Bootstrap starting.
I20260806 02:52:55.922577 6041 tablet_bootstrap.cc:492] T 4effcda7f378455694b9781859a2f7d7 P ab9b2267b3dc4fc794f9a3f59345fc80: Bootstrap starting.
I20260806 02:52:55.923976 6039 tablet_bootstrap.cc:654] T 4effcda7f378455694b9781859a2f7d7 P 90c9940f7ea24833bb84133bf92d11df: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:55.924731 6040 tablet_bootstrap.cc:654] T 4effcda7f378455694b9781859a2f7d7 P 048e0815a5df436985507ef1fae29d80: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:55.925478 6039 log.cc:824] T 4effcda7f378455694b9781859a2f7d7 P 90c9940f7ea24833bb84133bf92d11df: Log is configured to *not* fsync() on all Append() calls
I20260806 02:52:55.925688 6041 tablet_bootstrap.cc:654] T 4effcda7f378455694b9781859a2f7d7 P ab9b2267b3dc4fc794f9a3f59345fc80: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:55.925997 6040 log.cc:824] T 4effcda7f378455694b9781859a2f7d7 P 048e0815a5df436985507ef1fae29d80: Log is configured to *not* fsync() on all Append() calls
I20260806 02:52:55.926855 6041 log.cc:824] T 4effcda7f378455694b9781859a2f7d7 P ab9b2267b3dc4fc794f9a3f59345fc80: Log is configured to *not* fsync() on all Append() calls
I20260806 02:52:55.928462 6039 tablet_bootstrap.cc:492] T 4effcda7f378455694b9781859a2f7d7 P 90c9940f7ea24833bb84133bf92d11df: No bootstrap required, opened a new log
I20260806 02:52:55.928720 6039 ts_tablet_manager.cc:1397] T 4effcda7f378455694b9781859a2f7d7 P 90c9940f7ea24833bb84133bf92d11df: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260806 02:52:55.929210 6041 tablet_bootstrap.cc:492] T 4effcda7f378455694b9781859a2f7d7 P ab9b2267b3dc4fc794f9a3f59345fc80: No bootstrap required, opened a new log
I20260806 02:52:55.929440 6041 ts_tablet_manager.cc:1397] T 4effcda7f378455694b9781859a2f7d7 P ab9b2267b3dc4fc794f9a3f59345fc80: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260806 02:52:55.932312 6040 tablet_bootstrap.cc:492] T 4effcda7f378455694b9781859a2f7d7 P 048e0815a5df436985507ef1fae29d80: No bootstrap required, opened a new log
I20260806 02:52:55.932578 6040 ts_tablet_manager.cc:1397] T 4effcda7f378455694b9781859a2f7d7 P 048e0815a5df436985507ef1fae29d80: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260806 02:52:55.936457 6039 raft_consensus.cc:348] T 4effcda7f378455694b9781859a2f7d7 P 90c9940f7ea24833bb84133bf92d11df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90c9940f7ea24833bb84133bf92d11df" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40117 } } peers { permanent_uuid: "048e0815a5df436985507ef1fae29d80" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38389 } } peers { permanent_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42013 } }
I20260806 02:52:55.936826 6039 raft_consensus.cc:374] T 4effcda7f378455694b9781859a2f7d7 P 90c9940f7ea24833bb84133bf92d11df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:55.936924 6039 raft_consensus.cc:729] T 4effcda7f378455694b9781859a2f7d7 P 90c9940f7ea24833bb84133bf92d11df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90c9940f7ea24833bb84133bf92d11df, State: Initialized, Role: FOLLOWER
I20260806 02:52:55.937503 6039 consensus_queue.cc:260] T 4effcda7f378455694b9781859a2f7d7 P 90c9940f7ea24833bb84133bf92d11df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90c9940f7ea24833bb84133bf92d11df" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40117 } } peers { permanent_uuid: "048e0815a5df436985507ef1fae29d80" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38389 } } peers { permanent_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42013 } }
I20260806 02:52:55.938624 5774 heartbeater.cc:503] Master 127.25.211.190:37245 was elected leader, sending a full tablet report...
I20260806 02:52:55.939519 6039 ts_tablet_manager.cc:1428] T 4effcda7f378455694b9781859a2f7d7 P 90c9940f7ea24833bb84133bf92d11df: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260806 02:52:55.939309 6041 raft_consensus.cc:348] T 4effcda7f378455694b9781859a2f7d7 P ab9b2267b3dc4fc794f9a3f59345fc80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90c9940f7ea24833bb84133bf92d11df" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40117 } } peers { permanent_uuid: "048e0815a5df436985507ef1fae29d80" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38389 } } peers { permanent_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42013 } }
I20260806 02:52:55.939772 6041 raft_consensus.cc:374] T 4effcda7f378455694b9781859a2f7d7 P ab9b2267b3dc4fc794f9a3f59345fc80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:55.939862 6041 raft_consensus.cc:729] T 4effcda7f378455694b9781859a2f7d7 P ab9b2267b3dc4fc794f9a3f59345fc80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab9b2267b3dc4fc794f9a3f59345fc80, State: Initialized, Role: FOLLOWER
I20260806 02:52:55.940510 6041 consensus_queue.cc:260] T 4effcda7f378455694b9781859a2f7d7 P ab9b2267b3dc4fc794f9a3f59345fc80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90c9940f7ea24833bb84133bf92d11df" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40117 } } peers { permanent_uuid: "048e0815a5df436985507ef1fae29d80" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38389 } } peers { permanent_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42013 } }
I20260806 02:52:55.942965 6040 raft_consensus.cc:348] T 4effcda7f378455694b9781859a2f7d7 P 048e0815a5df436985507ef1fae29d80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90c9940f7ea24833bb84133bf92d11df" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40117 } } peers { permanent_uuid: "048e0815a5df436985507ef1fae29d80" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38389 } } peers { permanent_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42013 } }
I20260806 02:52:55.943465 6040 raft_consensus.cc:374] T 4effcda7f378455694b9781859a2f7d7 P 048e0815a5df436985507ef1fae29d80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:55.943578 6040 raft_consensus.cc:729] T 4effcda7f378455694b9781859a2f7d7 P 048e0815a5df436985507ef1fae29d80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 048e0815a5df436985507ef1fae29d80, State: Initialized, Role: FOLLOWER
W20260806 02:52:55.943668 5775 tablet.cc:2369] T 4effcda7f378455694b9781859a2f7d7 P 90c9940f7ea24833bb84133bf92d11df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:52:55.943887 6041 ts_tablet_manager.cc:1428] T 4effcda7f378455694b9781859a2f7d7 P ab9b2267b3dc4fc794f9a3f59345fc80: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260806 02:52:55.944243 6040 consensus_queue.cc:260] T 4effcda7f378455694b9781859a2f7d7 P 048e0815a5df436985507ef1fae29d80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90c9940f7ea24833bb84133bf92d11df" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40117 } } peers { permanent_uuid: "048e0815a5df436985507ef1fae29d80" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38389 } } peers { permanent_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42013 } }
I20260806 02:52:55.945384 6001 heartbeater.cc:503] Master 127.25.211.190:37245 was elected leader, sending a full tablet report...
I20260806 02:52:55.946102 6040 ts_tablet_manager.cc:1428] T 4effcda7f378455694b9781859a2f7d7 P 048e0815a5df436985507ef1fae29d80: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
W20260806 02:52:56.024325 6002 tablet.cc:2369] T 4effcda7f378455694b9781859a2f7d7 P 048e0815a5df436985507ef1fae29d80: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:52:56.123847 6051 raft_consensus.cc:482] T 4effcda7f378455694b9781859a2f7d7 P 048e0815a5df436985507ef1fae29d80 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:52:56.124217 6051 raft_consensus.cc:504] T 4effcda7f378455694b9781859a2f7d7 P 048e0815a5df436985507ef1fae29d80 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90c9940f7ea24833bb84133bf92d11df" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40117 } } peers { permanent_uuid: "048e0815a5df436985507ef1fae29d80" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38389 } } peers { permanent_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42013 } }
W20260806 02:52:56.125358 5560 tablet.cc:2369] T 4effcda7f378455694b9781859a2f7d7 P ab9b2267b3dc4fc794f9a3f59345fc80: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:52:56.126075 6051 leader_election.cc:288] T 4effcda7f378455694b9781859a2f7d7 P 048e0815a5df436985507ef1fae29d80 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90c9940f7ea24833bb84133bf92d11df (127.25.211.130:40117), ab9b2267b3dc4fc794f9a3f59345fc80 (127.25.211.129:42013)
I20260806 02:52:56.132863 6037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:56.126662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:48697 (local address 127.25.211.129:42013)
0806 02:52:56.126882 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:56.126886 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:56.127084 (+ 198us) server_negotiation.cc:407] Connection header received
0806 02:52:56.127315 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:56.127320 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:56.127401 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:56.127513 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:52:56.127950 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:56.128857 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:56.130847 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:56.131901 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:56.132034 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:56.132094 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:56.132478 (+ 384us) server_negotiation.cc:299] Negotiation successful
0806 02:52:56.132591 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 02:52:56.132915 5826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:56.126593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:42013 (local address 127.25.211.131:48697)
0806 02:52:56.126948 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:56.126968 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:56.127149 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:56.127546 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:56.127551 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:56.127589 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:52:56.127799 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:56.127812 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:56.129024 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:56.129030 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:56.130652 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:56.130666 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:56.132085 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:56.132091 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:56.132304 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:56.132341 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 02:52:56.132449 (+ 108us) client_negotiation.cc:240] Negotiation successful
0806 02:52:56.132601 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":1}
I20260806 02:52:56.133880 6030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:56.126515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:40125 (local address 127.25.211.130:40117)
0806 02:52:56.126742 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:56.126749 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:56.127552 (+ 803us) server_negotiation.cc:407] Connection header received
0806 02:52:56.127730 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:56.127736 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:56.127811 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:56.127908 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:52:56.128516 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:56.129470 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:56.131483 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:56.132496 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:56.132671 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:56.132757 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:56.133470 (+ 713us) server_negotiation.cc:299] Negotiation successful
0806 02:52:56.133614 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 02:52:56.133898 6052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:56.126774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.130:40117 (local address 127.25.211.131:40125)
0806 02:52:56.127433 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:56.127450 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:56.127570 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:56.127946 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:56.127950 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:56.127976 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:52:56.128315 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:56.128328 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:56.129640 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:56.129644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:56.131324 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:56.131340 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:56.132742 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:56.132748 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:56.132929 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:56.133305 (+ 376us) client_negotiation.cc:769] Sending connection context
0806 02:52:56.133410 (+ 105us) client_negotiation.cc:240] Negotiation successful
0806 02:52:56.133550 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":315,"threads_started":1}
I20260806 02:52:56.134126 5484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4effcda7f378455694b9781859a2f7d7" candidate_uuid: "048e0815a5df436985507ef1fae29d80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" is_pre_election: true
I20260806 02:52:56.134814 5484 raft_consensus.cc:2454] T 4effcda7f378455694b9781859a2f7d7 P ab9b2267b3dc4fc794f9a3f59345fc80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 048e0815a5df436985507ef1fae29d80 in term 0.
I20260806 02:52:56.135161 5702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4effcda7f378455694b9781859a2f7d7" candidate_uuid: "048e0815a5df436985507ef1fae29d80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90c9940f7ea24833bb84133bf92d11df" is_pre_election: true
I20260806 02:52:56.135588 5806 leader_election.cc:302] T 4effcda7f378455694b9781859a2f7d7 P 048e0815a5df436985507ef1fae29d80 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 048e0815a5df436985507ef1fae29d80, ab9b2267b3dc4fc794f9a3f59345fc80; no voters:
I20260806 02:52:56.135733 5702 raft_consensus.cc:2454] T 4effcda7f378455694b9781859a2f7d7 P 90c9940f7ea24833bb84133bf92d11df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 048e0815a5df436985507ef1fae29d80 in term 0.
I20260806 02:52:56.136072 6051 raft_consensus.cc:2781] T 4effcda7f378455694b9781859a2f7d7 P 048e0815a5df436985507ef1fae29d80 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:52:56.136217 6051 raft_consensus.cc:482] T 4effcda7f378455694b9781859a2f7d7 P 048e0815a5df436985507ef1fae29d80 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:52:56.136334 6051 raft_consensus.cc:3037] T 4effcda7f378455694b9781859a2f7d7 P 048e0815a5df436985507ef1fae29d80 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:56.138805 6051 raft_consensus.cc:504] T 4effcda7f378455694b9781859a2f7d7 P 048e0815a5df436985507ef1fae29d80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90c9940f7ea24833bb84133bf92d11df" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40117 } } peers { permanent_uuid: "048e0815a5df436985507ef1fae29d80" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38389 } } peers { permanent_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42013 } }
I20260806 02:52:56.139516 6051 leader_election.cc:288] T 4effcda7f378455694b9781859a2f7d7 P 048e0815a5df436985507ef1fae29d80 [CANDIDATE]: Term 1 election: Requested vote from peers 90c9940f7ea24833bb84133bf92d11df (127.25.211.130:40117), ab9b2267b3dc4fc794f9a3f59345fc80 (127.25.211.129:42013)
I20260806 02:52:56.140178 5702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4effcda7f378455694b9781859a2f7d7" candidate_uuid: "048e0815a5df436985507ef1fae29d80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90c9940f7ea24833bb84133bf92d11df"
I20260806 02:52:56.140182 5484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4effcda7f378455694b9781859a2f7d7" candidate_uuid: "048e0815a5df436985507ef1fae29d80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80"
I20260806 02:52:56.140400 5484 raft_consensus.cc:3037] T 4effcda7f378455694b9781859a2f7d7 P ab9b2267b3dc4fc794f9a3f59345fc80 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:56.140383 5702 raft_consensus.cc:3037] T 4effcda7f378455694b9781859a2f7d7 P 90c9940f7ea24833bb84133bf92d11df [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:56.142789 5484 raft_consensus.cc:2454] T 4effcda7f378455694b9781859a2f7d7 P ab9b2267b3dc4fc794f9a3f59345fc80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 048e0815a5df436985507ef1fae29d80 in term 1.
I20260806 02:52:56.142808 5702 raft_consensus.cc:2454] T 4effcda7f378455694b9781859a2f7d7 P 90c9940f7ea24833bb84133bf92d11df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 048e0815a5df436985507ef1fae29d80 in term 1.
I20260806 02:52:56.143306 5806 leader_election.cc:302] T 4effcda7f378455694b9781859a2f7d7 P 048e0815a5df436985507ef1fae29d80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 048e0815a5df436985507ef1fae29d80, ab9b2267b3dc4fc794f9a3f59345fc80; no voters:
I20260806 02:52:56.143666 6051 raft_consensus.cc:2781] T 4effcda7f378455694b9781859a2f7d7 P 048e0815a5df436985507ef1fae29d80 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:52:56.144102 6051 raft_consensus.cc:686] T 4effcda7f378455694b9781859a2f7d7 P 048e0815a5df436985507ef1fae29d80 [term 1 LEADER]: Becoming Leader. State: Replica: 048e0815a5df436985507ef1fae29d80, State: Running, Role: LEADER
I20260806 02:52:56.144610 6051 consensus_queue.cc:237] T 4effcda7f378455694b9781859a2f7d7 P 048e0815a5df436985507ef1fae29d80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90c9940f7ea24833bb84133bf92d11df" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40117 } } peers { permanent_uuid: "048e0815a5df436985507ef1fae29d80" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38389 } } peers { permanent_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42013 } }
I20260806 02:52:56.149905 5255 catalog_manager.cc:5495] T 4effcda7f378455694b9781859a2f7d7 P 048e0815a5df436985507ef1fae29d80 reported cstate change: term changed from 0 to 1, leader changed from <none> to 048e0815a5df436985507ef1fae29d80 (127.25.211.131). New cstate: current_term: 1 leader_uuid: "048e0815a5df436985507ef1fae29d80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90c9940f7ea24833bb84133bf92d11df" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "048e0815a5df436985507ef1fae29d80" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42013 } health_report { overall_health: UNKNOWN } } }
I20260806 02:52:56.205734 6031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:56.199439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37794 (local address 127.25.211.131:38389)
0806 02:52:56.199630 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:56.199636 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:56.199659 (+ 23us) server_negotiation.cc:407] Connection header received
0806 02:52:56.199791 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:56.199797 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:56.199907 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:56.200094 (+ 187us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:52:56.200536 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:56.201610 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:56.203248 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:56.203701 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:56.203846 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:56.204443 (+ 597us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:52:56.205087 (+ 644us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:52:56.205219 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:56.205365 (+ 146us) server_negotiation.cc:299] Negotiation successful
0806 02:52:56.205448 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 02:52:56.214704 5484 raft_consensus.cc:1261] T 4effcda7f378455694b9781859a2f7d7 P ab9b2267b3dc4fc794f9a3f59345fc80 [term 1 FOLLOWER]: Refusing update from remote peer 048e0815a5df436985507ef1fae29d80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:52:56.214700 5702 raft_consensus.cc:1261] T 4effcda7f378455694b9781859a2f7d7 P 90c9940f7ea24833bb84133bf92d11df [term 1 FOLLOWER]: Refusing update from remote peer 048e0815a5df436985507ef1fae29d80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:52:56.215757 6054 consensus_queue.cc:1035] T 4effcda7f378455694b9781859a2f7d7 P 048e0815a5df436985507ef1fae29d80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90c9940f7ea24833bb84133bf92d11df" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:52:56.216178 6051 consensus_queue.cc:1035] T 4effcda7f378455694b9781859a2f7d7 P 048e0815a5df436985507ef1fae29d80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:52:56.228317 6059 mvcc.cc:205] Tried to move back new op lower bound from 7315393643364585472 to 7315393643092668416. Current Snapshot: MvccSnapshot[applied={T|T < 7315393643364585472}]
I20260806 02:52:56.228750 6060 mvcc.cc:205] Tried to move back new op lower bound from 7315393643364585472 to 7315393643092668416. Current Snapshot: MvccSnapshot[applied={T|T < 7315393643364585472}]
I20260806 02:52:56.245931 5254 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:52:56.246426 5254 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:52:56.250869 5254 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8c91b83144bf42b38480be74a574bb17: Sending DeleteTablet for 3 replicas of tablet 4effcda7f378455694b9781859a2f7d7
I20260806 02:52:56.252242 5672 tablet_service.cc:1514] Processing DeleteTablet for tablet 4effcda7f378455694b9781859a2f7d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:52:56 UTC) from {username='oryx', principal='oryx/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:59784
I20260806 02:52:56.252272 5893 tablet_service.cc:1514] Processing DeleteTablet for tablet 4effcda7f378455694b9781859a2f7d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:52:56 UTC) from {username='oryx', principal='oryx/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:37790
I20260806 02:52:56.252570 5458 tablet_service.cc:1514] Processing DeleteTablet for tablet 4effcda7f378455694b9781859a2f7d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:52:56 UTC) from {username='oryx', principal='oryx/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:34414
I20260806 02:52:56.254988 5254 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56614:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:52:56.255846 5254 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:52:56.263120 6082 tablet_replica.cc:330] T 4effcda7f378455694b9781859a2f7d7 P ab9b2267b3dc4fc794f9a3f59345fc80: stopping tablet replica
I20260806 02:52:56.263684 6082 raft_consensus.cc:2229] T 4effcda7f378455694b9781859a2f7d7 P ab9b2267b3dc4fc794f9a3f59345fc80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:52:56.266768 5458 tablet_service.cc:1467] Processing CreateTablet for tablet 53f99c1fb10043f0b7c4bd0932fd4362 (DEFAULT_TABLE table=test-table [id=4b6b50951dfa437d9fb75a83e27ecd41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:52:56.266877 5671 tablet_service.cc:1467] Processing CreateTablet for tablet 53f99c1fb10043f0b7c4bd0932fd4362 (DEFAULT_TABLE table=test-table [id=4b6b50951dfa437d9fb75a83e27ecd41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:52:56.267273 5458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53f99c1fb10043f0b7c4bd0932fd4362. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:56.267520 5671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53f99c1fb10043f0b7c4bd0932fd4362. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:56.267967 6082 raft_consensus.cc:2258] T 4effcda7f378455694b9781859a2f7d7 P ab9b2267b3dc4fc794f9a3f59345fc80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:52:56.273417 6041 tablet_bootstrap.cc:492] T 53f99c1fb10043f0b7c4bd0932fd4362 P ab9b2267b3dc4fc794f9a3f59345fc80: Bootstrap starting.
I20260806 02:52:56.275117 6041 tablet_bootstrap.cc:654] T 53f99c1fb10043f0b7c4bd0932fd4362 P ab9b2267b3dc4fc794f9a3f59345fc80: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:56.276383 6039 tablet_bootstrap.cc:492] T 53f99c1fb10043f0b7c4bd0932fd4362 P 90c9940f7ea24833bb84133bf92d11df: Bootstrap starting.
I20260806 02:52:56.278299 6039 tablet_bootstrap.cc:654] T 53f99c1fb10043f0b7c4bd0932fd4362 P 90c9940f7ea24833bb84133bf92d11df: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:56.278304 6082 ts_tablet_manager.cc:1905] T 4effcda7f378455694b9781859a2f7d7 P ab9b2267b3dc4fc794f9a3f59345fc80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:52:56.280280 5891 tablet_service.cc:1467] Processing CreateTablet for tablet 53f99c1fb10043f0b7c4bd0932fd4362 (DEFAULT_TABLE table=test-table [id=4b6b50951dfa437d9fb75a83e27ecd41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:52:56.280889 5891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53f99c1fb10043f0b7c4bd0932fd4362. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:56.288822 6040 tablet_bootstrap.cc:492] T 53f99c1fb10043f0b7c4bd0932fd4362 P 048e0815a5df436985507ef1fae29d80: Bootstrap starting.
I20260806 02:52:56.290668 6075 tablet_replica.cc:330] T 4effcda7f378455694b9781859a2f7d7 P 048e0815a5df436985507ef1fae29d80: stopping tablet replica
I20260806 02:52:56.290714 6040 tablet_bootstrap.cc:654] T 53f99c1fb10043f0b7c4bd0932fd4362 P 048e0815a5df436985507ef1fae29d80: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:56.291210 6075 raft_consensus.cc:2229] T 4effcda7f378455694b9781859a2f7d7 P 048e0815a5df436985507ef1fae29d80 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:52:56.291923 6075 raft_consensus.cc:2258] T 4effcda7f378455694b9781859a2f7d7 P 048e0815a5df436985507ef1fae29d80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:52:56.293210 6039 tablet_bootstrap.cc:492] T 53f99c1fb10043f0b7c4bd0932fd4362 P 90c9940f7ea24833bb84133bf92d11df: No bootstrap required, opened a new log
I20260806 02:52:56.293464 6039 ts_tablet_manager.cc:1397] T 53f99c1fb10043f0b7c4bd0932fd4362 P 90c9940f7ea24833bb84133bf92d11df: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.004s
I20260806 02:52:56.294448 6039 raft_consensus.cc:348] T 53f99c1fb10043f0b7c4bd0932fd4362 P 90c9940f7ea24833bb84133bf92d11df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42013 } } peers { permanent_uuid: "90c9940f7ea24833bb84133bf92d11df" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40117 } } peers { permanent_uuid: "048e0815a5df436985507ef1fae29d80" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38389 } }
I20260806 02:52:56.294811 6039 raft_consensus.cc:374] T 53f99c1fb10043f0b7c4bd0932fd4362 P 90c9940f7ea24833bb84133bf92d11df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:56.294967 6039 raft_consensus.cc:729] T 53f99c1fb10043f0b7c4bd0932fd4362 P 90c9940f7ea24833bb84133bf92d11df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90c9940f7ea24833bb84133bf92d11df, State: Initialized, Role: FOLLOWER
I20260806 02:52:56.295380 6039 consensus_queue.cc:260] T 53f99c1fb10043f0b7c4bd0932fd4362 P 90c9940f7ea24833bb84133bf92d11df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42013 } } peers { permanent_uuid: "90c9940f7ea24833bb84133bf92d11df" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40117 } } peers { permanent_uuid: "048e0815a5df436985507ef1fae29d80" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38389 } }
I20260806 02:52:56.296329 6082 ts_tablet_manager.cc:1918] T 4effcda7f378455694b9781859a2f7d7 P ab9b2267b3dc4fc794f9a3f59345fc80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:52:56.296517 6082 log.cc:1196] T 4effcda7f378455694b9781859a2f7d7 P ab9b2267b3dc4fc794f9a3f59345fc80: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/ts-0/wal/wals/4effcda7f378455694b9781859a2f7d7
I20260806 02:52:56.297067 6082 ts_tablet_manager.cc:1939] T 4effcda7f378455694b9781859a2f7d7 P ab9b2267b3dc4fc794f9a3f59345fc80: Deleting consensus metadata
I20260806 02:52:56.300316 6040 tablet_bootstrap.cc:492] T 53f99c1fb10043f0b7c4bd0932fd4362 P 048e0815a5df436985507ef1fae29d80: No bootstrap required, opened a new log
I20260806 02:52:56.300263 5235 catalog_manager.cc:4841] TS ab9b2267b3dc4fc794f9a3f59345fc80 (127.25.211.129:42013): tablet 4effcda7f378455694b9781859a2f7d7 (table test-table [id=bd81223bc03b450095387fe90f4c0121]) successfully deleted
I20260806 02:52:56.300483 6040 ts_tablet_manager.cc:1397] T 53f99c1fb10043f0b7c4bd0932fd4362 P 048e0815a5df436985507ef1fae29d80: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.006s
I20260806 02:52:56.300655 6074 tablet_replica.cc:330] T 4effcda7f378455694b9781859a2f7d7 P 90c9940f7ea24833bb84133bf92d11df: stopping tablet replica
I20260806 02:52:56.300895 6039 ts_tablet_manager.cc:1428] T 53f99c1fb10043f0b7c4bd0932fd4362 P 90c9940f7ea24833bb84133bf92d11df: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260806 02:52:56.301154 6074 raft_consensus.cc:2229] T 4effcda7f378455694b9781859a2f7d7 P 90c9940f7ea24833bb84133bf92d11df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:52:56.301365 6040 raft_consensus.cc:348] T 53f99c1fb10043f0b7c4bd0932fd4362 P 048e0815a5df436985507ef1fae29d80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42013 } } peers { permanent_uuid: "90c9940f7ea24833bb84133bf92d11df" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40117 } } peers { permanent_uuid: "048e0815a5df436985507ef1fae29d80" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38389 } }
I20260806 02:52:56.301620 6040 raft_consensus.cc:374] T 53f99c1fb10043f0b7c4bd0932fd4362 P 048e0815a5df436985507ef1fae29d80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:56.301738 6040 raft_consensus.cc:729] T 53f99c1fb10043f0b7c4bd0932fd4362 P 048e0815a5df436985507ef1fae29d80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 048e0815a5df436985507ef1fae29d80, State: Initialized, Role: FOLLOWER
I20260806 02:52:56.301739 6074 raft_consensus.cc:2258] T 4effcda7f378455694b9781859a2f7d7 P 90c9940f7ea24833bb84133bf92d11df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:52:56.301985 6040 consensus_queue.cc:260] T 53f99c1fb10043f0b7c4bd0932fd4362 P 048e0815a5df436985507ef1fae29d80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42013 } } peers { permanent_uuid: "90c9940f7ea24833bb84133bf92d11df" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40117 } } peers { permanent_uuid: "048e0815a5df436985507ef1fae29d80" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38389 } }
I20260806 02:52:56.303012 6040 ts_tablet_manager.cc:1428] T 53f99c1fb10043f0b7c4bd0932fd4362 P 048e0815a5df436985507ef1fae29d80: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260806 02:52:56.304721 6041 tablet_bootstrap.cc:492] T 53f99c1fb10043f0b7c4bd0932fd4362 P ab9b2267b3dc4fc794f9a3f59345fc80: No bootstrap required, opened a new log
I20260806 02:52:56.304911 6041 ts_tablet_manager.cc:1397] T 53f99c1fb10043f0b7c4bd0932fd4362 P ab9b2267b3dc4fc794f9a3f59345fc80: Time spent bootstrapping tablet: real 0.032s user 0.000s sys 0.010s
I20260806 02:52:56.305373 6074 ts_tablet_manager.cc:1905] T 4effcda7f378455694b9781859a2f7d7 P 90c9940f7ea24833bb84133bf92d11df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:52:56.305830 6041 raft_consensus.cc:348] T 53f99c1fb10043f0b7c4bd0932fd4362 P ab9b2267b3dc4fc794f9a3f59345fc80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42013 } } peers { permanent_uuid: "90c9940f7ea24833bb84133bf92d11df" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40117 } } peers { permanent_uuid: "048e0815a5df436985507ef1fae29d80" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38389 } }
I20260806 02:52:56.306134 6041 raft_consensus.cc:374] T 53f99c1fb10043f0b7c4bd0932fd4362 P ab9b2267b3dc4fc794f9a3f59345fc80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:56.306270 6041 raft_consensus.cc:729] T 53f99c1fb10043f0b7c4bd0932fd4362 P ab9b2267b3dc4fc794f9a3f59345fc80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab9b2267b3dc4fc794f9a3f59345fc80, State: Initialized, Role: FOLLOWER
I20260806 02:52:56.306603 6041 consensus_queue.cc:260] T 53f99c1fb10043f0b7c4bd0932fd4362 P ab9b2267b3dc4fc794f9a3f59345fc80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42013 } } peers { permanent_uuid: "90c9940f7ea24833bb84133bf92d11df" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40117 } } peers { permanent_uuid: "048e0815a5df436985507ef1fae29d80" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38389 } }
I20260806 02:52:56.307991 6041 ts_tablet_manager.cc:1428] T 53f99c1fb10043f0b7c4bd0932fd4362 P ab9b2267b3dc4fc794f9a3f59345fc80: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260806 02:52:56.309928 6075 ts_tablet_manager.cc:1905] T 4effcda7f378455694b9781859a2f7d7 P 048e0815a5df436985507ef1fae29d80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:52:56.314610 6074 ts_tablet_manager.cc:1918] T 4effcda7f378455694b9781859a2f7d7 P 90c9940f7ea24833bb84133bf92d11df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:52:56.314934 6074 log.cc:1196] T 4effcda7f378455694b9781859a2f7d7 P 90c9940f7ea24833bb84133bf92d11df: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/ts-1/wal/wals/4effcda7f378455694b9781859a2f7d7
I20260806 02:52:56.315668 6074 ts_tablet_manager.cc:1939] T 4effcda7f378455694b9781859a2f7d7 P 90c9940f7ea24833bb84133bf92d11df: Deleting consensus metadata
I20260806 02:52:56.317096 6075 ts_tablet_manager.cc:1918] T 4effcda7f378455694b9781859a2f7d7 P 048e0815a5df436985507ef1fae29d80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:52:56.317318 6075 log.cc:1196] T 4effcda7f378455694b9781859a2f7d7 P 048e0815a5df436985507ef1fae29d80: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/ts-2/wal/wals/4effcda7f378455694b9781859a2f7d7
I20260806 02:52:56.317821 6075 ts_tablet_manager.cc:1939] T 4effcda7f378455694b9781859a2f7d7 P 048e0815a5df436985507ef1fae29d80: Deleting consensus metadata
I20260806 02:52:56.318066 5234 catalog_manager.cc:4841] TS 90c9940f7ea24833bb84133bf92d11df (127.25.211.130:40117): tablet 4effcda7f378455694b9781859a2f7d7 (table test-table [id=bd81223bc03b450095387fe90f4c0121]) successfully deleted
I20260806 02:52:56.319607 5238 catalog_manager.cc:4841] TS 048e0815a5df436985507ef1fae29d80 (127.25.211.131:38389): tablet 4effcda7f378455694b9781859a2f7d7 (table test-table [id=bd81223bc03b450095387fe90f4c0121]) successfully deleted
I20260806 02:52:56.362483 6048 raft_consensus.cc:482] T 53f99c1fb10043f0b7c4bd0932fd4362 P 90c9940f7ea24833bb84133bf92d11df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:52:56.362735 6048 raft_consensus.cc:504] T 53f99c1fb10043f0b7c4bd0932fd4362 P 90c9940f7ea24833bb84133bf92d11df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42013 } } peers { permanent_uuid: "90c9940f7ea24833bb84133bf92d11df" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40117 } } peers { permanent_uuid: "048e0815a5df436985507ef1fae29d80" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38389 } }
I20260806 02:52:56.364383 6048 leader_election.cc:288] T 53f99c1fb10043f0b7c4bd0932fd4362 P 90c9940f7ea24833bb84133bf92d11df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab9b2267b3dc4fc794f9a3f59345fc80 (127.25.211.129:42013), 048e0815a5df436985507ef1fae29d80 (127.25.211.131:38389)
I20260806 02:52:56.373600 6037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:56.365259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:37051 (local address 127.25.211.129:42013)
0806 02:52:56.365436 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:56.365441 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:56.365985 (+ 544us) server_negotiation.cc:407] Connection header received
0806 02:52:56.366099 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:56.366104 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:56.366202 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:56.366289 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:52:56.367083 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:56.368153 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:56.371007 (+ 2854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:56.372322 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:56.372522 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:56.372606 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:56.373131 (+ 525us) server_negotiation.cc:299] Negotiation successful
0806 02:52:56.373294 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 02:52:56.373634 6031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:56.364794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:38487 (local address 127.25.211.131:38389)
0806 02:52:56.365012 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:56.365019 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:56.365513 (+ 494us) server_negotiation.cc:407] Connection header received
0806 02:52:56.365930 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:56.365936 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:56.366021 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:56.366123 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:52:56.366845 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:56.368056 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:56.371289 (+ 3233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:56.372595 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:56.372748 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:56.372829 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:56.373208 (+ 379us) server_negotiation.cc:299] Negotiation successful
0806 02:52:56.373347 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260806 02:52:56.373664 6090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:56.364829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:42013 (local address 127.25.211.130:37051)
0806 02:52:56.365875 (+ 1046us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:56.365891 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:56.365991 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:56.366359 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:56.366364 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:56.366410 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:52:56.366897 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:56.366909 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:56.368303 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:56.368307 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:56.370770 (+ 2463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:56.370792 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:56.372619 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:56.372628 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:56.372907 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:56.372948 (+ 41us) client_negotiation.cc:769] Sending connection context
0806 02:52:56.373064 (+ 116us) client_negotiation.cc:240] Negotiation successful
0806 02:52:56.373228 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":913,"mutex_wait_us":183,"thread_start_us":708,"threads_started":1}
I20260806 02:52:56.374472 6089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:56.364742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.131:38389 (local address 127.25.211.130:38487)
0806 02:52:56.365397 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:56.365416 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:56.365690 (+ 274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:56.366263 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:56.366268 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:56.366294 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:52:56.366646 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:56.366656 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:56.368239 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:56.368245 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:56.371051 (+ 2806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:56.371072 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:56.372824 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:56.372831 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:56.373014 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:56.373045 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 02:52:56.373916 (+ 871us) client_negotiation.cc:240] Negotiation successful
0806 02:52:56.374077 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"mutex_wait_us":77,"thread_start_us":397,"threads_started":1}
I20260806 02:52:56.374640 5484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53f99c1fb10043f0b7c4bd0932fd4362" candidate_uuid: "90c9940f7ea24833bb84133bf92d11df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" is_pre_election: true
I20260806 02:52:56.374938 5484 raft_consensus.cc:2454] T 53f99c1fb10043f0b7c4bd0932fd4362 P ab9b2267b3dc4fc794f9a3f59345fc80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90c9940f7ea24833bb84133bf92d11df in term 0.
I20260806 02:52:56.375241 6066 raft_consensus.cc:482] T 53f99c1fb10043f0b7c4bd0932fd4362 P 048e0815a5df436985507ef1fae29d80 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:52:56.375459 6066 raft_consensus.cc:504] T 53f99c1fb10043f0b7c4bd0932fd4362 P 048e0815a5df436985507ef1fae29d80 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42013 } } peers { permanent_uuid: "90c9940f7ea24833bb84133bf92d11df" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40117 } } peers { permanent_uuid: "048e0815a5df436985507ef1fae29d80" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38389 } }
I20260806 02:52:56.375491 5924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53f99c1fb10043f0b7c4bd0932fd4362" candidate_uuid: "90c9940f7ea24833bb84133bf92d11df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "048e0815a5df436985507ef1fae29d80" is_pre_election: true
I20260806 02:52:56.376087 5924 raft_consensus.cc:2454] T 53f99c1fb10043f0b7c4bd0932fd4362 P 048e0815a5df436985507ef1fae29d80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90c9940f7ea24833bb84133bf92d11df in term 0.
I20260806 02:52:56.376637 6066 leader_election.cc:288] T 53f99c1fb10043f0b7c4bd0932fd4362 P 048e0815a5df436985507ef1fae29d80 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab9b2267b3dc4fc794f9a3f59345fc80 (127.25.211.129:42013), 90c9940f7ea24833bb84133bf92d11df (127.25.211.130:40117)
I20260806 02:52:56.377600 5597 leader_election.cc:302] T 53f99c1fb10043f0b7c4bd0932fd4362 P 90c9940f7ea24833bb84133bf92d11df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 048e0815a5df436985507ef1fae29d80, 90c9940f7ea24833bb84133bf92d11df; no voters:
I20260806 02:52:56.377723 5484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53f99c1fb10043f0b7c4bd0932fd4362" candidate_uuid: "048e0815a5df436985507ef1fae29d80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" is_pre_election: true
I20260806 02:52:56.377983 5484 raft_consensus.cc:2454] T 53f99c1fb10043f0b7c4bd0932fd4362 P ab9b2267b3dc4fc794f9a3f59345fc80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 048e0815a5df436985507ef1fae29d80 in term 0.
I20260806 02:52:56.378189 5702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53f99c1fb10043f0b7c4bd0932fd4362" candidate_uuid: "048e0815a5df436985507ef1fae29d80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90c9940f7ea24833bb84133bf92d11df" is_pre_election: true
I20260806 02:52:56.378455 5702 raft_consensus.cc:2454] T 53f99c1fb10043f0b7c4bd0932fd4362 P 90c9940f7ea24833bb84133bf92d11df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 048e0815a5df436985507ef1fae29d80 in term 0.
I20260806 02:52:56.378515 5806 leader_election.cc:302] T 53f99c1fb10043f0b7c4bd0932fd4362 P 048e0815a5df436985507ef1fae29d80 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 048e0815a5df436985507ef1fae29d80, ab9b2267b3dc4fc794f9a3f59345fc80; no voters:
I20260806 02:52:56.378738 6048 raft_consensus.cc:2781] T 53f99c1fb10043f0b7c4bd0932fd4362 P 90c9940f7ea24833bb84133bf92d11df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:52:56.378849 6048 raft_consensus.cc:482] T 53f99c1fb10043f0b7c4bd0932fd4362 P 90c9940f7ea24833bb84133bf92d11df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:52:56.378947 6048 raft_consensus.cc:3037] T 53f99c1fb10043f0b7c4bd0932fd4362 P 90c9940f7ea24833bb84133bf92d11df [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:56.381336 6048 raft_consensus.cc:504] T 53f99c1fb10043f0b7c4bd0932fd4362 P 90c9940f7ea24833bb84133bf92d11df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42013 } } peers { permanent_uuid: "90c9940f7ea24833bb84133bf92d11df" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40117 } } peers { permanent_uuid: "048e0815a5df436985507ef1fae29d80" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38389 } }
I20260806 02:52:56.382411 6066 raft_consensus.cc:2781] T 53f99c1fb10043f0b7c4bd0932fd4362 P 048e0815a5df436985507ef1fae29d80 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:52:56.382580 6066 raft_consensus.cc:482] T 53f99c1fb10043f0b7c4bd0932fd4362 P 048e0815a5df436985507ef1fae29d80 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:52:56.382544 5484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53f99c1fb10043f0b7c4bd0932fd4362" candidate_uuid: "90c9940f7ea24833bb84133bf92d11df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80"
I20260806 02:52:56.382686 6066 raft_consensus.cc:3037] T 53f99c1fb10043f0b7c4bd0932fd4362 P 048e0815a5df436985507ef1fae29d80 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:56.382748 6048 leader_election.cc:288] T 53f99c1fb10043f0b7c4bd0932fd4362 P 90c9940f7ea24833bb84133bf92d11df [CANDIDATE]: Term 1 election: Requested vote from peers ab9b2267b3dc4fc794f9a3f59345fc80 (127.25.211.129:42013), 048e0815a5df436985507ef1fae29d80 (127.25.211.131:38389)
I20260806 02:52:56.382790 5484 raft_consensus.cc:3037] T 53f99c1fb10043f0b7c4bd0932fd4362 P ab9b2267b3dc4fc794f9a3f59345fc80 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:56.383665 5924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53f99c1fb10043f0b7c4bd0932fd4362" candidate_uuid: "90c9940f7ea24833bb84133bf92d11df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "048e0815a5df436985507ef1fae29d80"
I20260806 02:52:56.385761 6066 raft_consensus.cc:504] T 53f99c1fb10043f0b7c4bd0932fd4362 P 048e0815a5df436985507ef1fae29d80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42013 } } peers { permanent_uuid: "90c9940f7ea24833bb84133bf92d11df" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40117 } } peers { permanent_uuid: "048e0815a5df436985507ef1fae29d80" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38389 } }
I20260806 02:52:56.385900 5484 raft_consensus.cc:2454] T 53f99c1fb10043f0b7c4bd0932fd4362 P ab9b2267b3dc4fc794f9a3f59345fc80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90c9940f7ea24833bb84133bf92d11df in term 1.
I20260806 02:52:56.386262 5924 raft_consensus.cc:2379] T 53f99c1fb10043f0b7c4bd0932fd4362 P 048e0815a5df436985507ef1fae29d80 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 90c9940f7ea24833bb84133bf92d11df in current term 1: Already voted for candidate 048e0815a5df436985507ef1fae29d80 in this term.
I20260806 02:52:56.386570 5595 leader_election.cc:302] T 53f99c1fb10043f0b7c4bd0932fd4362 P 90c9940f7ea24833bb84133bf92d11df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90c9940f7ea24833bb84133bf92d11df, ab9b2267b3dc4fc794f9a3f59345fc80; no voters:
I20260806 02:52:56.387112 6066 leader_election.cc:288] T 53f99c1fb10043f0b7c4bd0932fd4362 P 048e0815a5df436985507ef1fae29d80 [CANDIDATE]: Term 1 election: Requested vote from peers ab9b2267b3dc4fc794f9a3f59345fc80 (127.25.211.129:42013), 90c9940f7ea24833bb84133bf92d11df (127.25.211.130:40117)
I20260806 02:52:56.387485 5702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53f99c1fb10043f0b7c4bd0932fd4362" candidate_uuid: "048e0815a5df436985507ef1fae29d80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90c9940f7ea24833bb84133bf92d11df"
I20260806 02:52:56.387722 5702 raft_consensus.cc:2379] T 53f99c1fb10043f0b7c4bd0932fd4362 P 90c9940f7ea24833bb84133bf92d11df [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 048e0815a5df436985507ef1fae29d80 in current term 1: Already voted for candidate 90c9940f7ea24833bb84133bf92d11df in this term.
I20260806 02:52:56.388190 6048 raft_consensus.cc:2781] T 53f99c1fb10043f0b7c4bd0932fd4362 P 90c9940f7ea24833bb84133bf92d11df [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:52:56.388633 5484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53f99c1fb10043f0b7c4bd0932fd4362" candidate_uuid: "048e0815a5df436985507ef1fae29d80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80"
I20260806 02:52:56.389111 5484 raft_consensus.cc:2379] T 53f99c1fb10043f0b7c4bd0932fd4362 P ab9b2267b3dc4fc794f9a3f59345fc80 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 048e0815a5df436985507ef1fae29d80 in current term 1: Already voted for candidate 90c9940f7ea24833bb84133bf92d11df in this term.
I20260806 02:52:56.389340 6048 raft_consensus.cc:686] T 53f99c1fb10043f0b7c4bd0932fd4362 P 90c9940f7ea24833bb84133bf92d11df [term 1 LEADER]: Becoming Leader. State: Replica: 90c9940f7ea24833bb84133bf92d11df, State: Running, Role: LEADER
I20260806 02:52:56.389792 5806 leader_election.cc:302] T 53f99c1fb10043f0b7c4bd0932fd4362 P 048e0815a5df436985507ef1fae29d80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 048e0815a5df436985507ef1fae29d80; no voters: 90c9940f7ea24833bb84133bf92d11df, ab9b2267b3dc4fc794f9a3f59345fc80
I20260806 02:52:56.389866 6048 consensus_queue.cc:237] T 53f99c1fb10043f0b7c4bd0932fd4362 P 90c9940f7ea24833bb84133bf92d11df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42013 } } peers { permanent_uuid: "90c9940f7ea24833bb84133bf92d11df" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40117 } } peers { permanent_uuid: "048e0815a5df436985507ef1fae29d80" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38389 } }
I20260806 02:52:56.390283 6066 raft_consensus.cc:2726] T 53f99c1fb10043f0b7c4bd0932fd4362 P 048e0815a5df436985507ef1fae29d80 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260806 02:52:56.394011 5254 catalog_manager.cc:5495] T 53f99c1fb10043f0b7c4bd0932fd4362 P 90c9940f7ea24833bb84133bf92d11df reported cstate change: term changed from 0 to 1, leader changed from <none> to 90c9940f7ea24833bb84133bf92d11df (127.25.211.130). New cstate: current_term: 1 leader_uuid: "90c9940f7ea24833bb84133bf92d11df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90c9940f7ea24833bb84133bf92d11df" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "048e0815a5df436985507ef1fae29d80" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38389 } health_report { overall_health: UNKNOWN } } }
Aug 06 02:52:56 dist-test-slave-zj8q krb5kdc[5196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984773, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.211.190@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM
I20260806 02:52:56.448375 6100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:56.432939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56616 (local address 127.25.211.190:37245)
0806 02:52:56.434207 (+ 1268us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:56.434213 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:56.434235 (+ 22us) server_negotiation.cc:407] Connection header received
0806 02:52:56.434284 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:56.434288 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:56.434357 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:56.434449 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:56.435608 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:56.436398 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:56.437763 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:56.438235 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:56.438348 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:56.442222 (+ 3874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:56.442229 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:56.442233 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:56.442265 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:56.444805 (+ 2540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:56.445913 (+ 1108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:56.445917 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:56.445919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:56.446100 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:56.446610 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:56.446615 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:56.446618 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:56.447007 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:56.447120 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:56.447701 (+ 581us) server_negotiation.cc:299] Negotiation successful
0806 02:52:56.448068 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":162,"server-negotiator.queue_time_us":1076,"thread_start_us":826,"threads_started":1}
I20260806 02:52:56.448398 6099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:56.432832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:37245 (local address 127.0.0.1:56616)
0806 02:52:56.433706 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:56.433734 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:56.434013 (+ 279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:56.434524 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:56.434532 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:56.435245 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:52:56.435444 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:56.435454 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:56.436550 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:56.436554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:56.437613 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:56.437627 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:56.438399 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:56.438405 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:56.438568 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:56.439297 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:52:56.439315 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:52:56.441944 (+ 2629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:52:56.444958 (+ 3014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:56.444974 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:56.444983 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:56.445685 (+ 702us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:56.446263 (+ 578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:52:56.446268 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:52:56.446272 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:52:56.446447 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:52:56.447159 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:52:56.447168 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:52:56.447390 (+ 222us) client_negotiation.cc:769] Sending connection context
0806 02:52:56.447579 (+ 189us) client_negotiation.cc:240] Negotiation successful
0806 02:52:56.447927 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"thread_start_us":525,"threads_started":1}
I20260806 02:52:56.454705 5254 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:56616:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 02:52:56.456015 5254 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:52:56.463668 5458 tablet_service.cc:1467] Processing CreateTablet for tablet 38897b5bbbd047fe8d9f63899286eac9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=24652d7809d94f49acc943b2a63d6ab4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 02:52:56.463778 5672 tablet_service.cc:1467] Processing CreateTablet for tablet 38897b5bbbd047fe8d9f63899286eac9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=24652d7809d94f49acc943b2a63d6ab4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 02:52:56.464290 5458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38897b5bbbd047fe8d9f63899286eac9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:56.464365 5672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38897b5bbbd047fe8d9f63899286eac9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:56.466878 5893 tablet_service.cc:1467] Processing CreateTablet for tablet 38897b5bbbd047fe8d9f63899286eac9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=24652d7809d94f49acc943b2a63d6ab4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 02:52:56.467530 5893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38897b5bbbd047fe8d9f63899286eac9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:56.470979 6039 tablet_bootstrap.cc:492] T 38897b5bbbd047fe8d9f63899286eac9 P 90c9940f7ea24833bb84133bf92d11df: Bootstrap starting.
I20260806 02:52:56.470984 6041 tablet_bootstrap.cc:492] T 38897b5bbbd047fe8d9f63899286eac9 P ab9b2267b3dc4fc794f9a3f59345fc80: Bootstrap starting.
I20260806 02:52:56.472711 6039 tablet_bootstrap.cc:654] T 38897b5bbbd047fe8d9f63899286eac9 P 90c9940f7ea24833bb84133bf92d11df: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:56.472714 6041 tablet_bootstrap.cc:654] T 38897b5bbbd047fe8d9f63899286eac9 P ab9b2267b3dc4fc794f9a3f59345fc80: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:56.478209 6040 tablet_bootstrap.cc:492] T 38897b5bbbd047fe8d9f63899286eac9 P 048e0815a5df436985507ef1fae29d80: Bootstrap starting.
I20260806 02:52:56.480062 6040 tablet_bootstrap.cc:654] T 38897b5bbbd047fe8d9f63899286eac9 P 048e0815a5df436985507ef1fae29d80: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:56.480295 6041 tablet_bootstrap.cc:492] T 38897b5bbbd047fe8d9f63899286eac9 P ab9b2267b3dc4fc794f9a3f59345fc80: No bootstrap required, opened a new log
I20260806 02:52:56.480484 6041 ts_tablet_manager.cc:1397] T 38897b5bbbd047fe8d9f63899286eac9 P ab9b2267b3dc4fc794f9a3f59345fc80: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260806 02:52:56.481302 6041 raft_consensus.cc:348] T 38897b5bbbd047fe8d9f63899286eac9 P ab9b2267b3dc4fc794f9a3f59345fc80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42013 } } peers { permanent_uuid: "90c9940f7ea24833bb84133bf92d11df" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40117 } } peers { permanent_uuid: "048e0815a5df436985507ef1fae29d80" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38389 } }
I20260806 02:52:56.481693 6041 raft_consensus.cc:374] T 38897b5bbbd047fe8d9f63899286eac9 P ab9b2267b3dc4fc794f9a3f59345fc80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:56.481776 6041 raft_consensus.cc:729] T 38897b5bbbd047fe8d9f63899286eac9 P ab9b2267b3dc4fc794f9a3f59345fc80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab9b2267b3dc4fc794f9a3f59345fc80, State: Initialized, Role: FOLLOWER
I20260806 02:52:56.482093 6041 consensus_queue.cc:260] T 38897b5bbbd047fe8d9f63899286eac9 P ab9b2267b3dc4fc794f9a3f59345fc80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42013 } } peers { permanent_uuid: "90c9940f7ea24833bb84133bf92d11df" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40117 } } peers { permanent_uuid: "048e0815a5df436985507ef1fae29d80" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38389 } }
I20260806 02:52:56.482621 6039 tablet_bootstrap.cc:492] T 38897b5bbbd047fe8d9f63899286eac9 P 90c9940f7ea24833bb84133bf92d11df: No bootstrap required, opened a new log
I20260806 02:52:56.482877 6039 ts_tablet_manager.cc:1397] T 38897b5bbbd047fe8d9f63899286eac9 P 90c9940f7ea24833bb84133bf92d11df: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.004s
I20260806 02:52:56.482690 6050 tablet_replica.cc:439] T 38897b5bbbd047fe8d9f63899286eac9 P ab9b2267b3dc4fc794f9a3f59345fc80: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42013 } } peers { permanent_uuid: "90c9940f7ea24833bb84133bf92d11df" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40117 } } peers { permanent_uuid: "048e0815a5df436985507ef1fae29d80" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38389 } } }
I20260806 02:52:56.483055 6050 tablet_replica.cc:442] T 38897b5bbbd047fe8d9f63899286eac9 P ab9b2267b3dc4fc794f9a3f59345fc80: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:52:56.483822 6039 raft_consensus.cc:348] T 38897b5bbbd047fe8d9f63899286eac9 P 90c9940f7ea24833bb84133bf92d11df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42013 } } peers { permanent_uuid: "90c9940f7ea24833bb84133bf92d11df" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40117 } } peers { permanent_uuid: "048e0815a5df436985507ef1fae29d80" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38389 } }
I20260806 02:52:56.484140 6039 raft_consensus.cc:374] T 38897b5bbbd047fe8d9f63899286eac9 P 90c9940f7ea24833bb84133bf92d11df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:56.484273 6039 raft_consensus.cc:729] T 38897b5bbbd047fe8d9f63899286eac9 P 90c9940f7ea24833bb84133bf92d11df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90c9940f7ea24833bb84133bf92d11df, State: Initialized, Role: FOLLOWER
I20260806 02:52:56.484930 6039 consensus_queue.cc:260] T 38897b5bbbd047fe8d9f63899286eac9 P 90c9940f7ea24833bb84133bf92d11df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42013 } } peers { permanent_uuid: "90c9940f7ea24833bb84133bf92d11df" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40117 } } peers { permanent_uuid: "048e0815a5df436985507ef1fae29d80" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38389 } }
I20260806 02:52:56.485440 6048 tablet_replica.cc:439] T 38897b5bbbd047fe8d9f63899286eac9 P 90c9940f7ea24833bb84133bf92d11df: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42013 } } peers { permanent_uuid: "90c9940f7ea24833bb84133bf92d11df" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40117 } } peers { permanent_uuid: "048e0815a5df436985507ef1fae29d80" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38389 } } }
I20260806 02:52:56.485787 6048 tablet_replica.cc:442] T 38897b5bbbd047fe8d9f63899286eac9 P 90c9940f7ea24833bb84133bf92d11df: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:52:56.486367 6039 ts_tablet_manager.cc:1428] T 38897b5bbbd047fe8d9f63899286eac9 P 90c9940f7ea24833bb84133bf92d11df: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260806 02:52:56.486979 6041 ts_tablet_manager.cc:1428] T 38897b5bbbd047fe8d9f63899286eac9 P ab9b2267b3dc4fc794f9a3f59345fc80: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260806 02:52:56.491366 6040 tablet_bootstrap.cc:492] T 38897b5bbbd047fe8d9f63899286eac9 P 048e0815a5df436985507ef1fae29d80: No bootstrap required, opened a new log
I20260806 02:52:56.491757 6040 ts_tablet_manager.cc:1397] T 38897b5bbbd047fe8d9f63899286eac9 P 048e0815a5df436985507ef1fae29d80: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.004s
I20260806 02:52:56.492662 6040 raft_consensus.cc:348] T 38897b5bbbd047fe8d9f63899286eac9 P 048e0815a5df436985507ef1fae29d80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42013 } } peers { permanent_uuid: "90c9940f7ea24833bb84133bf92d11df" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40117 } } peers { permanent_uuid: "048e0815a5df436985507ef1fae29d80" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38389 } }
I20260806 02:52:56.492890 6040 raft_consensus.cc:374] T 38897b5bbbd047fe8d9f63899286eac9 P 048e0815a5df436985507ef1fae29d80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:56.492974 6040 raft_consensus.cc:729] T 38897b5bbbd047fe8d9f63899286eac9 P 048e0815a5df436985507ef1fae29d80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 048e0815a5df436985507ef1fae29d80, State: Initialized, Role: FOLLOWER
I20260806 02:52:56.493232 6040 consensus_queue.cc:260] T 38897b5bbbd047fe8d9f63899286eac9 P 048e0815a5df436985507ef1fae29d80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42013 } } peers { permanent_uuid: "90c9940f7ea24833bb84133bf92d11df" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40117 } } peers { permanent_uuid: "048e0815a5df436985507ef1fae29d80" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38389 } }
I20260806 02:52:56.493680 6066 tablet_replica.cc:439] T 38897b5bbbd047fe8d9f63899286eac9 P 048e0815a5df436985507ef1fae29d80: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42013 } } peers { permanent_uuid: "90c9940f7ea24833bb84133bf92d11df" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40117 } } peers { permanent_uuid: "048e0815a5df436985507ef1fae29d80" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38389 } } }
I20260806 02:52:56.493891 6040 ts_tablet_manager.cc:1428] T 38897b5bbbd047fe8d9f63899286eac9 P 048e0815a5df436985507ef1fae29d80: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260806 02:52:56.494043 6066 tablet_replica.cc:442] T 38897b5bbbd047fe8d9f63899286eac9 P 048e0815a5df436985507ef1fae29d80: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:52:56.524266 6048 raft_consensus.cc:482] T 38897b5bbbd047fe8d9f63899286eac9 P 90c9940f7ea24833bb84133bf92d11df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:52:56.524555 6048 raft_consensus.cc:504] T 38897b5bbbd047fe8d9f63899286eac9 P 90c9940f7ea24833bb84133bf92d11df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42013 } } peers { permanent_uuid: "90c9940f7ea24833bb84133bf92d11df" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40117 } } peers { permanent_uuid: "048e0815a5df436985507ef1fae29d80" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38389 } }
I20260806 02:52:56.525326 6048 leader_election.cc:288] T 38897b5bbbd047fe8d9f63899286eac9 P 90c9940f7ea24833bb84133bf92d11df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab9b2267b3dc4fc794f9a3f59345fc80 (127.25.211.129:42013), 048e0815a5df436985507ef1fae29d80 (127.25.211.131:38389)
I20260806 02:52:56.525950 5484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38897b5bbbd047fe8d9f63899286eac9" candidate_uuid: "90c9940f7ea24833bb84133bf92d11df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" is_pre_election: true
I20260806 02:52:56.526115 5924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38897b5bbbd047fe8d9f63899286eac9" candidate_uuid: "90c9940f7ea24833bb84133bf92d11df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "048e0815a5df436985507ef1fae29d80" is_pre_election: true
I20260806 02:52:56.526295 5484 raft_consensus.cc:2454] T 38897b5bbbd047fe8d9f63899286eac9 P ab9b2267b3dc4fc794f9a3f59345fc80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90c9940f7ea24833bb84133bf92d11df in term 0.
I20260806 02:52:56.526409 5924 raft_consensus.cc:2454] T 38897b5bbbd047fe8d9f63899286eac9 P 048e0815a5df436985507ef1fae29d80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90c9940f7ea24833bb84133bf92d11df in term 0.
I20260806 02:52:56.526738 5595 leader_election.cc:302] T 38897b5bbbd047fe8d9f63899286eac9 P 90c9940f7ea24833bb84133bf92d11df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90c9940f7ea24833bb84133bf92d11df, ab9b2267b3dc4fc794f9a3f59345fc80; no voters:
I20260806 02:52:56.527064 6048 raft_consensus.cc:2781] T 38897b5bbbd047fe8d9f63899286eac9 P 90c9940f7ea24833bb84133bf92d11df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:52:56.527174 6048 raft_consensus.cc:482] T 38897b5bbbd047fe8d9f63899286eac9 P 90c9940f7ea24833bb84133bf92d11df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:52:56.527297 6048 raft_consensus.cc:3037] T 38897b5bbbd047fe8d9f63899286eac9 P 90c9940f7ea24833bb84133bf92d11df [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:56.530297 6048 raft_consensus.cc:504] T 38897b5bbbd047fe8d9f63899286eac9 P 90c9940f7ea24833bb84133bf92d11df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42013 } } peers { permanent_uuid: "90c9940f7ea24833bb84133bf92d11df" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40117 } } peers { permanent_uuid: "048e0815a5df436985507ef1fae29d80" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38389 } }
I20260806 02:52:56.531018 6048 leader_election.cc:288] T 38897b5bbbd047fe8d9f63899286eac9 P 90c9940f7ea24833bb84133bf92d11df [CANDIDATE]: Term 1 election: Requested vote from peers ab9b2267b3dc4fc794f9a3f59345fc80 (127.25.211.129:42013), 048e0815a5df436985507ef1fae29d80 (127.25.211.131:38389)
I20260806 02:52:56.531507 5484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38897b5bbbd047fe8d9f63899286eac9" candidate_uuid: "90c9940f7ea24833bb84133bf92d11df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80"
I20260806 02:52:56.531561 5924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38897b5bbbd047fe8d9f63899286eac9" candidate_uuid: "90c9940f7ea24833bb84133bf92d11df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "048e0815a5df436985507ef1fae29d80"
I20260806 02:52:56.531747 5484 raft_consensus.cc:3037] T 38897b5bbbd047fe8d9f63899286eac9 P ab9b2267b3dc4fc794f9a3f59345fc80 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:56.531780 5924 raft_consensus.cc:3037] T 38897b5bbbd047fe8d9f63899286eac9 P 048e0815a5df436985507ef1fae29d80 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:56.533797 5484 raft_consensus.cc:2454] T 38897b5bbbd047fe8d9f63899286eac9 P ab9b2267b3dc4fc794f9a3f59345fc80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90c9940f7ea24833bb84133bf92d11df in term 1.
I20260806 02:52:56.534250 5595 leader_election.cc:302] T 38897b5bbbd047fe8d9f63899286eac9 P 90c9940f7ea24833bb84133bf92d11df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90c9940f7ea24833bb84133bf92d11df, ab9b2267b3dc4fc794f9a3f59345fc80; no voters:
I20260806 02:52:56.534549 6048 raft_consensus.cc:2781] T 38897b5bbbd047fe8d9f63899286eac9 P 90c9940f7ea24833bb84133bf92d11df [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:52:56.534876 6048 raft_consensus.cc:686] T 38897b5bbbd047fe8d9f63899286eac9 P 90c9940f7ea24833bb84133bf92d11df [term 1 LEADER]: Becoming Leader. State: Replica: 90c9940f7ea24833bb84133bf92d11df, State: Running, Role: LEADER
I20260806 02:52:56.535048 5924 raft_consensus.cc:2454] T 38897b5bbbd047fe8d9f63899286eac9 P 048e0815a5df436985507ef1fae29d80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90c9940f7ea24833bb84133bf92d11df in term 1.
I20260806 02:52:56.535135 6048 consensus_queue.cc:237] T 38897b5bbbd047fe8d9f63899286eac9 P 90c9940f7ea24833bb84133bf92d11df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42013 } } peers { permanent_uuid: "90c9940f7ea24833bb84133bf92d11df" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40117 } } peers { permanent_uuid: "048e0815a5df436985507ef1fae29d80" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38389 } }
I20260806 02:52:56.536916 6091 tablet_replica.cc:439] T 38897b5bbbd047fe8d9f63899286eac9 P 90c9940f7ea24833bb84133bf92d11df: TxnStatusTablet state changed. Reason: New leader 90c9940f7ea24833bb84133bf92d11df. Latest consensus state: current_term: 1 leader_uuid: "90c9940f7ea24833bb84133bf92d11df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42013 } } peers { permanent_uuid: "90c9940f7ea24833bb84133bf92d11df" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40117 } } peers { permanent_uuid: "048e0815a5df436985507ef1fae29d80" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38389 } } }
I20260806 02:52:56.537371 6091 tablet_replica.cc:442] T 38897b5bbbd047fe8d9f63899286eac9 P 90c9940f7ea24833bb84133bf92d11df: This TxnStatusTablet replica's current role is: LEADER
I20260806 02:52:56.538105 6107 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 02:52:56.538340 6107 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 02:52:56.538968 5254 catalog_manager.cc:5495] T 38897b5bbbd047fe8d9f63899286eac9 P 90c9940f7ea24833bb84133bf92d11df reported cstate change: term changed from 0 to 1, leader changed from <none> to 90c9940f7ea24833bb84133bf92d11df (127.25.211.130). New cstate: current_term: 1 leader_uuid: "90c9940f7ea24833bb84133bf92d11df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90c9940f7ea24833bb84133bf92d11df" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "048e0815a5df436985507ef1fae29d80" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38389 } health_report { overall_health: UNKNOWN } } }
I20260806 02:52:56.579209 6030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:56.570443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59798 (local address 127.25.211.130:40117)
0806 02:52:56.571346 (+ 903us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:56.571352 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:56.571373 (+ 21us) server_negotiation.cc:407] Connection header received
0806 02:52:56.571477 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:56.571483 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:56.571615 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:56.571785 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:52:56.572326 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:56.573696 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:56.575882 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:56.576592 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:56.576736 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:56.577411 (+ 675us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:52:56.578317 (+ 906us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:52:56.578481 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:56.578752 (+ 271us) server_negotiation.cc:299] Negotiation successful
0806 02:52:56.578857 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":728}
I20260806 02:52:56.579205 6099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:56.570460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.130:40117 (local address 127.0.0.1:59798)
0806 02:52:56.570772 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:56.570796 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:56.570984 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:56.571875 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:56.571881 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:56.571896 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:52:56.572138 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:56.572152 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:56.573891 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:56.573897 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:56.575650 (+ 1753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:56.575670 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:56.576813 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:56.576819 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:56.577036 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:56.577128 (+ 92us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:52:56.578548 (+ 1420us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:52:56.578581 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 02:52:56.578679 (+ 98us) client_negotiation.cc:240] Negotiation successful
0806 02:52:56.578869 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260806 02:52:56.584707 5484 raft_consensus.cc:1261] T 38897b5bbbd047fe8d9f63899286eac9 P ab9b2267b3dc4fc794f9a3f59345fc80 [term 1 FOLLOWER]: Refusing update from remote peer 90c9940f7ea24833bb84133bf92d11df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:52:56.585507 6091 consensus_queue.cc:1035] T 38897b5bbbd047fe8d9f63899286eac9 P 90c9940f7ea24833bb84133bf92d11df [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:52:56.585350 5924 raft_consensus.cc:1261] T 38897b5bbbd047fe8d9f63899286eac9 P 048e0815a5df436985507ef1fae29d80 [term 1 FOLLOWER]: Refusing update from remote peer 90c9940f7ea24833bb84133bf92d11df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:52:56.588655 6091 consensus_queue.cc:1035] T 38897b5bbbd047fe8d9f63899286eac9 P 90c9940f7ea24833bb84133bf92d11df [LEADER]: Connected to new peer: Peer: permanent_uuid: "048e0815a5df436985507ef1fae29d80" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:52:56.590443 6050 tablet_replica.cc:439] T 38897b5bbbd047fe8d9f63899286eac9 P ab9b2267b3dc4fc794f9a3f59345fc80: TxnStatusTablet state changed. Reason: New leader 90c9940f7ea24833bb84133bf92d11df. Latest consensus state: current_term: 1 leader_uuid: "90c9940f7ea24833bb84133bf92d11df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42013 } } peers { permanent_uuid: "90c9940f7ea24833bb84133bf92d11df" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40117 } } peers { permanent_uuid: "048e0815a5df436985507ef1fae29d80" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38389 } } }
I20260806 02:52:56.590725 6050 tablet_replica.cc:442] T 38897b5bbbd047fe8d9f63899286eac9 P ab9b2267b3dc4fc794f9a3f59345fc80: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:52:56.595077 6061 mvcc.cc:205] Tried to move back new op lower bound from 7315393644881920000 to 7315393644690128896. Current Snapshot: MvccSnapshot[applied={T|T < 7315393644881920000 or (T in {7315393644881920000})}]
I20260806 02:52:56.595497 6050 tablet_replica.cc:439] T 38897b5bbbd047fe8d9f63899286eac9 P ab9b2267b3dc4fc794f9a3f59345fc80: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "90c9940f7ea24833bb84133bf92d11df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42013 } } peers { permanent_uuid: "90c9940f7ea24833bb84133bf92d11df" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40117 } } peers { permanent_uuid: "048e0815a5df436985507ef1fae29d80" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38389 } } }
I20260806 02:52:56.595747 6050 tablet_replica.cc:442] T 38897b5bbbd047fe8d9f63899286eac9 P ab9b2267b3dc4fc794f9a3f59345fc80: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:52:56.596853 6048 tablet_replica.cc:439] T 38897b5bbbd047fe8d9f63899286eac9 P 90c9940f7ea24833bb84133bf92d11df: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "90c9940f7ea24833bb84133bf92d11df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42013 } } peers { permanent_uuid: "90c9940f7ea24833bb84133bf92d11df" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40117 } } peers { permanent_uuid: "048e0815a5df436985507ef1fae29d80" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38389 } } }
I20260806 02:52:56.597204 6048 tablet_replica.cc:442] T 38897b5bbbd047fe8d9f63899286eac9 P 90c9940f7ea24833bb84133bf92d11df: This TxnStatusTablet replica's current role is: LEADER
I20260806 02:52:56.605659 6066 tablet_replica.cc:439] T 38897b5bbbd047fe8d9f63899286eac9 P 048e0815a5df436985507ef1fae29d80: TxnStatusTablet state changed. Reason: New leader 90c9940f7ea24833bb84133bf92d11df. Latest consensus state: current_term: 1 leader_uuid: "90c9940f7ea24833bb84133bf92d11df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42013 } } peers { permanent_uuid: "90c9940f7ea24833bb84133bf92d11df" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40117 } } peers { permanent_uuid: "048e0815a5df436985507ef1fae29d80" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38389 } } }
I20260806 02:52:56.606050 6066 tablet_replica.cc:442] T 38897b5bbbd047fe8d9f63899286eac9 P 048e0815a5df436985507ef1fae29d80: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:52:56.610667 6066 tablet_replica.cc:439] T 38897b5bbbd047fe8d9f63899286eac9 P 048e0815a5df436985507ef1fae29d80: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "90c9940f7ea24833bb84133bf92d11df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42013 } } peers { permanent_uuid: "90c9940f7ea24833bb84133bf92d11df" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40117 } } peers { permanent_uuid: "048e0815a5df436985507ef1fae29d80" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38389 } } }
I20260806 02:52:56.610983 6066 tablet_replica.cc:442] T 38897b5bbbd047fe8d9f63899286eac9 P 048e0815a5df436985507ef1fae29d80: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 02:52:56.612241 6091 tablet_replica.cc:439] T 38897b5bbbd047fe8d9f63899286eac9 P 90c9940f7ea24833bb84133bf92d11df: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "90c9940f7ea24833bb84133bf92d11df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42013 } } peers { permanent_uuid: "90c9940f7ea24833bb84133bf92d11df" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40117 } } peers { permanent_uuid: "048e0815a5df436985507ef1fae29d80" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38389 } } }
I20260806 02:52:56.612506 6091 tablet_replica.cc:442] T 38897b5bbbd047fe8d9f63899286eac9 P 90c9940f7ea24833bb84133bf92d11df: This TxnStatusTablet replica's current role is: LEADER
I20260806 02:52:56.622323 6030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:56.608736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59802 (local address 127.25.211.130:40117)
0806 02:52:56.609374 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:56.609380 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:56.609399 (+ 19us) server_negotiation.cc:407] Connection header received
0806 02:52:56.609459 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:56.609464 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:56.609541 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:56.613678 (+ 4137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:52:56.613861 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:56.615080 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:56.617417 (+ 2337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:56.618206 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:56.620718 (+ 2512us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:56.620924 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:52:56.621063 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:52:56.621850 (+ 787us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:56.621900 (+ 50us) server_negotiation.cc:299] Negotiation successful
0806 02:52:56.622002 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499}
I20260806 02:52:56.658267 6030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:56.649427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:44449 (local address 127.25.211.130:40117)
0806 02:52:56.649677 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:56.649689 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:56.650862 (+ 1173us) server_negotiation.cc:407] Connection header received
0806 02:52:56.651254 (+ 392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:56.651277 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:56.651367 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:56.651490 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:52:56.652245 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:56.653433 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:56.655754 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:56.656515 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:56.656653 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:56.657280 (+ 627us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:52:56.657455 (+ 175us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:52:56.657550 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:56.657768 (+ 218us) server_negotiation.cc:299] Negotiation successful
0806 02:52:56.657865 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":112}
I20260806 02:52:56.658425 6115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:56.649500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.130:40117 (local address 127.25.211.130:44449)
0806 02:52:56.650701 (+ 1201us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:56.650726 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:56.650940 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:56.651527 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:56.651533 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:56.651546 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:52:56.652025 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:56.652044 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:56.653613 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:56.653619 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:56.655557 (+ 1938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:56.655576 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:56.656772 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:56.656780 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:56.656958 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:56.657062 (+ 104us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:52:56.657595 (+ 533us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:52:56.657609 (+ 14us) client_negotiation.cc:769] Sending connection context
0806 02:52:56.657713 (+ 104us) client_negotiation.cc:240] Negotiation successful
0806 02:52:56.657894 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":967,"mutex_wait_us":24,"thread_start_us":780,"threads_started":1}
I20260806 02:52:56.674659 5484 raft_consensus.cc:1261] T 53f99c1fb10043f0b7c4bd0932fd4362 P ab9b2267b3dc4fc794f9a3f59345fc80 [term 1 FOLLOWER]: Refusing update from remote peer 90c9940f7ea24833bb84133bf92d11df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:52:56.674677 5924 raft_consensus.cc:1261] T 53f99c1fb10043f0b7c4bd0932fd4362 P 048e0815a5df436985507ef1fae29d80 [term 1 FOLLOWER]: Refusing update from remote peer 90c9940f7ea24833bb84133bf92d11df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:52:56.675453 6048 consensus_queue.cc:1035] T 53f99c1fb10043f0b7c4bd0932fd4362 P 90c9940f7ea24833bb84133bf92d11df [LEADER]: Connected to new peer: Peer: permanent_uuid: "048e0815a5df436985507ef1fae29d80" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:52:56.676100 6112 consensus_queue.cc:1035] T 53f99c1fb10043f0b7c4bd0932fd4362 P 90c9940f7ea24833bb84133bf92d11df [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:52:56.781481 5254 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:52:56.781785 5254 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:52:56.785938 5254 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8c91b83144bf42b38480be74a574bb17: Sending DeleteTablet for 3 replicas of tablet 53f99c1fb10043f0b7c4bd0932fd4362
I20260806 02:52:56.787272 5893 tablet_service.cc:1514] Processing DeleteTablet for tablet 53f99c1fb10043f0b7c4bd0932fd4362 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:52:56 UTC) from {username='oryx', principal='oryx/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:37790
I20260806 02:52:56.787623 6075 tablet_replica.cc:330] T 53f99c1fb10043f0b7c4bd0932fd4362 P 048e0815a5df436985507ef1fae29d80: stopping tablet replica
I20260806 02:52:56.787819 5458 tablet_service.cc:1514] Processing DeleteTablet for tablet 53f99c1fb10043f0b7c4bd0932fd4362 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:52:56 UTC) from {username='oryx', principal='oryx/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:34414
I20260806 02:52:56.787856 6075 raft_consensus.cc:2229] T 53f99c1fb10043f0b7c4bd0932fd4362 P 048e0815a5df436985507ef1fae29d80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:52:56.788065 6075 raft_consensus.cc:2258] T 53f99c1fb10043f0b7c4bd0932fd4362 P 048e0815a5df436985507ef1fae29d80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:52:56.788117 6082 tablet_replica.cc:330] T 53f99c1fb10043f0b7c4bd0932fd4362 P ab9b2267b3dc4fc794f9a3f59345fc80: stopping tablet replica
I20260806 02:52:56.788347 6082 raft_consensus.cc:2229] T 53f99c1fb10043f0b7c4bd0932fd4362 P ab9b2267b3dc4fc794f9a3f59345fc80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:52:56.788543 6082 raft_consensus.cc:2258] T 53f99c1fb10043f0b7c4bd0932fd4362 P ab9b2267b3dc4fc794f9a3f59345fc80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:52:56.789417 5671 tablet_service.cc:1514] Processing DeleteTablet for tablet 53f99c1fb10043f0b7c4bd0932fd4362 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:52:56 UTC) from {username='oryx', principal='oryx/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:59784
I20260806 02:52:56.789710 6074 tablet_replica.cc:330] T 53f99c1fb10043f0b7c4bd0932fd4362 P 90c9940f7ea24833bb84133bf92d11df: stopping tablet replica
I20260806 02:52:56.789916 6074 raft_consensus.cc:2229] T 53f99c1fb10043f0b7c4bd0932fd4362 P 90c9940f7ea24833bb84133bf92d11df [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:52:56.790432 6074 raft_consensus.cc:2258] T 53f99c1fb10043f0b7c4bd0932fd4362 P 90c9940f7ea24833bb84133bf92d11df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:52:56.790746 6075 ts_tablet_manager.cc:1905] T 53f99c1fb10043f0b7c4bd0932fd4362 P 048e0815a5df436985507ef1fae29d80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:52:56.791637 6074 ts_tablet_manager.cc:1905] T 53f99c1fb10043f0b7c4bd0932fd4362 P 90c9940f7ea24833bb84133bf92d11df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:52:56.794909 6082 ts_tablet_manager.cc:1905] T 53f99c1fb10043f0b7c4bd0932fd4362 P ab9b2267b3dc4fc794f9a3f59345fc80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:52:56.799806 6074 ts_tablet_manager.cc:1918] T 53f99c1fb10043f0b7c4bd0932fd4362 P 90c9940f7ea24833bb84133bf92d11df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:52:56.800081 6074 log.cc:1196] T 53f99c1fb10043f0b7c4bd0932fd4362 P 90c9940f7ea24833bb84133bf92d11df: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/ts-1/wal/wals/53f99c1fb10043f0b7c4bd0932fd4362
I20260806 02:52:56.800582 6074 ts_tablet_manager.cc:1939] T 53f99c1fb10043f0b7c4bd0932fd4362 P 90c9940f7ea24833bb84133bf92d11df: Deleting consensus metadata
I20260806 02:52:56.810498 5234 catalog_manager.cc:4841] TS 90c9940f7ea24833bb84133bf92d11df (127.25.211.130:40117): tablet 53f99c1fb10043f0b7c4bd0932fd4362 (table test-table [id=4b6b50951dfa437d9fb75a83e27ecd41]) successfully deleted
I20260806 02:52:56.813488 6075 ts_tablet_manager.cc:1918] T 53f99c1fb10043f0b7c4bd0932fd4362 P 048e0815a5df436985507ef1fae29d80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:52:56.813678 6082 ts_tablet_manager.cc:1918] T 53f99c1fb10043f0b7c4bd0932fd4362 P ab9b2267b3dc4fc794f9a3f59345fc80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:52:56.813880 6082 log.cc:1196] T 53f99c1fb10043f0b7c4bd0932fd4362 P ab9b2267b3dc4fc794f9a3f59345fc80: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/ts-0/wal/wals/53f99c1fb10043f0b7c4bd0932fd4362
I20260806 02:52:56.813900 6075 log.cc:1196] T 53f99c1fb10043f0b7c4bd0932fd4362 P 048e0815a5df436985507ef1fae29d80: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/ts-2/wal/wals/53f99c1fb10043f0b7c4bd0932fd4362
I20260806 02:52:56.814334 6082 ts_tablet_manager.cc:1939] T 53f99c1fb10043f0b7c4bd0932fd4362 P ab9b2267b3dc4fc794f9a3f59345fc80: Deleting consensus metadata
I20260806 02:52:56.814563 6075 ts_tablet_manager.cc:1939] T 53f99c1fb10043f0b7c4bd0932fd4362 P 048e0815a5df436985507ef1fae29d80: Deleting consensus metadata
I20260806 02:52:56.815661 5235 catalog_manager.cc:4841] TS ab9b2267b3dc4fc794f9a3f59345fc80 (127.25.211.129:42013): tablet 53f99c1fb10043f0b7c4bd0932fd4362 (table test-table [id=4b6b50951dfa437d9fb75a83e27ecd41]) successfully deleted
I20260806 02:52:56.816498 5238 catalog_manager.cc:4841] TS 048e0815a5df436985507ef1fae29d80 (127.25.211.131:38389): tablet 53f99c1fb10043f0b7c4bd0932fd4362 (table test-table [id=4b6b50951dfa437d9fb75a83e27ecd41]) successfully deleted
Aug 06 02:52:56 dist-test-slave-zj8q krb5kdc[5196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 02:52:56 dist-test-slave-zj8q krb5kdc[5196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:52:56 dist-test-slave-zj8q krb5kdc[5196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
W20260806 02:52:56.878557 6100 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:52:56.865415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56624 (local address 127.25.211.190:37245)
0806 02:52:56.865692 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:56.865699 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:56.865957 (+ 258us) server_negotiation.cc:407] Connection header received
0806 02:52:56.866227 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:56.866232 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:56.866303 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:56.866392 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:56.868279 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:56.869114 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:56.871065 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:56.871613 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:56.871739 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:56.878282 (+ 6543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56624: BlockingRecv error: recv got EOF from 127.0.0.1:56624 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 06 02:52:56 dist-test-slave-zj8q krb5kdc[5196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM
I20260806 02:52:56.898926 6100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:56.881969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56628 (local address 127.25.211.190:37245)
0806 02:52:56.882222 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:56.882229 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:56.882571 (+ 342us) server_negotiation.cc:407] Connection header received
0806 02:52:56.882768 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:56.882774 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:56.882860 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:56.882962 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:56.884281 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:56.885089 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:56.886554 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:56.887050 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:56.887177 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:56.891636 (+ 4459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:56.891644 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:56.891649 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:56.891706 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:56.895443 (+ 3737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:56.896491 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:56.896497 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:56.896501 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:56.896677 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:56.897208 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:56.897223 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:56.897225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:56.897600 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:56.897885 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:56.898283 (+ 398us) server_negotiation.cc:299] Negotiation successful
0806 02:52:56.898547 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 02:52:56.902853 5254 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56628:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:52:56.903523 5254 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:52:56.911298 5458 tablet_service.cc:1467] Processing CreateTablet for tablet 999e47b3a8ac4a18bd2eb72234307852 (DEFAULT_TABLE table=test-table [id=4d76c4a9408440fdbf4d212b0ada5864]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:52:56.911875 5458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 999e47b3a8ac4a18bd2eb72234307852. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:56.917822 5893 tablet_service.cc:1467] Processing CreateTablet for tablet 999e47b3a8ac4a18bd2eb72234307852 (DEFAULT_TABLE table=test-table [id=4d76c4a9408440fdbf4d212b0ada5864]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:52:56.917872 5671 tablet_service.cc:1467] Processing CreateTablet for tablet 999e47b3a8ac4a18bd2eb72234307852 (DEFAULT_TABLE table=test-table [id=4d76c4a9408440fdbf4d212b0ada5864]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:52:56.918378 5893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 999e47b3a8ac4a18bd2eb72234307852. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:56.918404 5671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 999e47b3a8ac4a18bd2eb72234307852. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:56.920287 6041 tablet_bootstrap.cc:492] T 999e47b3a8ac4a18bd2eb72234307852 P ab9b2267b3dc4fc794f9a3f59345fc80: Bootstrap starting.
I20260806 02:52:56.922030 6041 tablet_bootstrap.cc:654] T 999e47b3a8ac4a18bd2eb72234307852 P ab9b2267b3dc4fc794f9a3f59345fc80: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:56.925171 6040 tablet_bootstrap.cc:492] T 999e47b3a8ac4a18bd2eb72234307852 P 048e0815a5df436985507ef1fae29d80: Bootstrap starting.
I20260806 02:52:56.925371 6039 tablet_bootstrap.cc:492] T 999e47b3a8ac4a18bd2eb72234307852 P 90c9940f7ea24833bb84133bf92d11df: Bootstrap starting.
I20260806 02:52:56.927093 6040 tablet_bootstrap.cc:654] T 999e47b3a8ac4a18bd2eb72234307852 P 048e0815a5df436985507ef1fae29d80: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:56.927088 6039 tablet_bootstrap.cc:654] T 999e47b3a8ac4a18bd2eb72234307852 P 90c9940f7ea24833bb84133bf92d11df: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:56.936934 6041 tablet_bootstrap.cc:492] T 999e47b3a8ac4a18bd2eb72234307852 P ab9b2267b3dc4fc794f9a3f59345fc80: No bootstrap required, opened a new log
I20260806 02:52:56.937152 6041 ts_tablet_manager.cc:1397] T 999e47b3a8ac4a18bd2eb72234307852 P ab9b2267b3dc4fc794f9a3f59345fc80: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.007s
I20260806 02:52:56.938460 6041 raft_consensus.cc:348] T 999e47b3a8ac4a18bd2eb72234307852 P ab9b2267b3dc4fc794f9a3f59345fc80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42013 } } peers { permanent_uuid: "90c9940f7ea24833bb84133bf92d11df" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40117 } } peers { permanent_uuid: "048e0815a5df436985507ef1fae29d80" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38389 } }
I20260806 02:52:56.938796 6041 raft_consensus.cc:374] T 999e47b3a8ac4a18bd2eb72234307852 P ab9b2267b3dc4fc794f9a3f59345fc80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:56.938907 6041 raft_consensus.cc:729] T 999e47b3a8ac4a18bd2eb72234307852 P ab9b2267b3dc4fc794f9a3f59345fc80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab9b2267b3dc4fc794f9a3f59345fc80, State: Initialized, Role: FOLLOWER
I20260806 02:52:56.939167 6041 consensus_queue.cc:260] T 999e47b3a8ac4a18bd2eb72234307852 P ab9b2267b3dc4fc794f9a3f59345fc80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42013 } } peers { permanent_uuid: "90c9940f7ea24833bb84133bf92d11df" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40117 } } peers { permanent_uuid: "048e0815a5df436985507ef1fae29d80" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38389 } }
I20260806 02:52:56.939742 6039 tablet_bootstrap.cc:492] T 999e47b3a8ac4a18bd2eb72234307852 P 90c9940f7ea24833bb84133bf92d11df: No bootstrap required, opened a new log
I20260806 02:52:56.939945 6039 ts_tablet_manager.cc:1397] T 999e47b3a8ac4a18bd2eb72234307852 P 90c9940f7ea24833bb84133bf92d11df: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20260806 02:52:56.940448 6041 ts_tablet_manager.cc:1428] T 999e47b3a8ac4a18bd2eb72234307852 P ab9b2267b3dc4fc794f9a3f59345fc80: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260806 02:52:56.940852 6039 raft_consensus.cc:348] T 999e47b3a8ac4a18bd2eb72234307852 P 90c9940f7ea24833bb84133bf92d11df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42013 } } peers { permanent_uuid: "90c9940f7ea24833bb84133bf92d11df" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40117 } } peers { permanent_uuid: "048e0815a5df436985507ef1fae29d80" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38389 } }
I20260806 02:52:56.941138 6039 raft_consensus.cc:374] T 999e47b3a8ac4a18bd2eb72234307852 P 90c9940f7ea24833bb84133bf92d11df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:56.941248 6039 raft_consensus.cc:729] T 999e47b3a8ac4a18bd2eb72234307852 P 90c9940f7ea24833bb84133bf92d11df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90c9940f7ea24833bb84133bf92d11df, State: Initialized, Role: FOLLOWER
I20260806 02:52:56.941392 6040 tablet_bootstrap.cc:492] T 999e47b3a8ac4a18bd2eb72234307852 P 048e0815a5df436985507ef1fae29d80: No bootstrap required, opened a new log
I20260806 02:52:56.941541 6040 ts_tablet_manager.cc:1397] T 999e47b3a8ac4a18bd2eb72234307852 P 048e0815a5df436985507ef1fae29d80: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.008s
I20260806 02:52:56.941509 6039 consensus_queue.cc:260] T 999e47b3a8ac4a18bd2eb72234307852 P 90c9940f7ea24833bb84133bf92d11df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42013 } } peers { permanent_uuid: "90c9940f7ea24833bb84133bf92d11df" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40117 } } peers { permanent_uuid: "048e0815a5df436985507ef1fae29d80" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38389 } }
I20260806 02:52:56.943110 6040 raft_consensus.cc:348] T 999e47b3a8ac4a18bd2eb72234307852 P 048e0815a5df436985507ef1fae29d80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42013 } } peers { permanent_uuid: "90c9940f7ea24833bb84133bf92d11df" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40117 } } peers { permanent_uuid: "048e0815a5df436985507ef1fae29d80" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38389 } }
I20260806 02:52:56.943385 6040 raft_consensus.cc:374] T 999e47b3a8ac4a18bd2eb72234307852 P 048e0815a5df436985507ef1fae29d80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:56.943486 6040 raft_consensus.cc:729] T 999e47b3a8ac4a18bd2eb72234307852 P 048e0815a5df436985507ef1fae29d80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 048e0815a5df436985507ef1fae29d80, State: Initialized, Role: FOLLOWER
I20260806 02:52:56.943704 6040 consensus_queue.cc:260] T 999e47b3a8ac4a18bd2eb72234307852 P 048e0815a5df436985507ef1fae29d80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42013 } } peers { permanent_uuid: "90c9940f7ea24833bb84133bf92d11df" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40117 } } peers { permanent_uuid: "048e0815a5df436985507ef1fae29d80" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38389 } }
I20260806 02:52:56.944278 6040 ts_tablet_manager.cc:1428] T 999e47b3a8ac4a18bd2eb72234307852 P 048e0815a5df436985507ef1fae29d80: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260806 02:52:56.945835 6039 ts_tablet_manager.cc:1428] T 999e47b3a8ac4a18bd2eb72234307852 P 90c9940f7ea24833bb84133bf92d11df: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20260806 02:52:57.176826 6050 raft_consensus.cc:482] T 999e47b3a8ac4a18bd2eb72234307852 P ab9b2267b3dc4fc794f9a3f59345fc80 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:52:57.177093 6050 raft_consensus.cc:504] T 999e47b3a8ac4a18bd2eb72234307852 P ab9b2267b3dc4fc794f9a3f59345fc80 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42013 } } peers { permanent_uuid: "90c9940f7ea24833bb84133bf92d11df" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40117 } } peers { permanent_uuid: "048e0815a5df436985507ef1fae29d80" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38389 } }
I20260806 02:52:57.179016 6050 leader_election.cc:288] T 999e47b3a8ac4a18bd2eb72234307852 P ab9b2267b3dc4fc794f9a3f59345fc80 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90c9940f7ea24833bb84133bf92d11df (127.25.211.130:40117), 048e0815a5df436985507ef1fae29d80 (127.25.211.131:38389)
I20260806 02:52:57.197577 6154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:57.179242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.131:38389 (local address 127.25.211.129:50977)
0806 02:52:57.189450 (+ 10208us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:57.189474 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:57.189662 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:57.190194 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:57.190200 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:57.190240 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:52:57.190521 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:57.190534 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:57.192151 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:57.192158 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:57.194465 (+ 2307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:57.194489 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:57.196355 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:57.196362 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:57.196563 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:57.196599 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 02:52:57.196683 (+ 84us) client_negotiation.cc:240] Negotiation successful
0806 02:52:57.196931 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10055,"thread_start_us":9989,"threads_started":1}
I20260806 02:52:57.198935 6157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:57.179864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:50977 (local address 127.25.211.131:38389)
0806 02:52:57.189215 (+ 9351us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:57.189222 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:57.189787 (+ 565us) server_negotiation.cc:407] Connection header received
0806 02:52:57.189871 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:57.189876 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:57.189955 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:57.190712 (+ 757us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:52:57.190865 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:57.191948 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:57.194719 (+ 2771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:57.195978 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:57.196151 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:57.196243 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:57.198474 (+ 2231us) server_negotiation.cc:299] Negotiation successful
0806 02:52:57.198637 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9189,"thread_start_us":9079,"threads_started":1}
I20260806 02:52:57.199775 5918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "999e47b3a8ac4a18bd2eb72234307852" candidate_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "048e0815a5df436985507ef1fae29d80" is_pre_election: true
I20260806 02:52:57.200063 5918 raft_consensus.cc:2454] T 999e47b3a8ac4a18bd2eb72234307852 P 048e0815a5df436985507ef1fae29d80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab9b2267b3dc4fc794f9a3f59345fc80 in term 0.
I20260806 02:52:57.200685 5386 leader_election.cc:302] T 999e47b3a8ac4a18bd2eb72234307852 P ab9b2267b3dc4fc794f9a3f59345fc80 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 048e0815a5df436985507ef1fae29d80, ab9b2267b3dc4fc794f9a3f59345fc80; no voters:
I20260806 02:52:57.201210 6050 raft_consensus.cc:2781] T 999e47b3a8ac4a18bd2eb72234307852 P ab9b2267b3dc4fc794f9a3f59345fc80 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:52:57.201372 6050 raft_consensus.cc:482] T 999e47b3a8ac4a18bd2eb72234307852 P ab9b2267b3dc4fc794f9a3f59345fc80 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:52:57.201495 6050 raft_consensus.cc:3037] T 999e47b3a8ac4a18bd2eb72234307852 P ab9b2267b3dc4fc794f9a3f59345fc80 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:57.204931 6050 raft_consensus.cc:504] T 999e47b3a8ac4a18bd2eb72234307852 P ab9b2267b3dc4fc794f9a3f59345fc80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42013 } } peers { permanent_uuid: "90c9940f7ea24833bb84133bf92d11df" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40117 } } peers { permanent_uuid: "048e0815a5df436985507ef1fae29d80" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38389 } }
I20260806 02:52:57.205735 6156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:57.179553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:48209 (local address 127.25.211.130:40117)
0806 02:52:57.187418 (+ 7865us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:57.187428 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:57.196139 (+ 8711us) server_negotiation.cc:407] Connection header received
0806 02:52:57.197931 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:57.197946 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:57.198046 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:57.198133 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:52:57.199096 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:57.200164 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:57.202748 (+ 2584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:57.204075 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:57.204247 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:57.204344 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:57.204993 (+ 649us) server_negotiation.cc:299] Negotiation successful
0806 02:52:57.205337 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7657,"thread_start_us":7458,"threads_started":1}
I20260806 02:52:57.206594 6155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:57.179243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.130:40117 (local address 127.25.211.129:48209)
0806 02:52:57.196008 (+ 16765us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:57.196036 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:57.196501 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:57.198521 (+ 2020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:57.198526 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:57.198556 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:52:57.198853 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:57.198863 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:57.200361 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:57.200366 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:57.202541 (+ 2175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:57.202564 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:57.204575 (+ 2011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:57.204582 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:57.204756 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:57.204794 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 02:52:57.206032 (+ 1238us) client_negotiation.cc:240] Negotiation successful
0806 02:52:57.206205 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16611,"mutex_wait_us":156,"thread_start_us":16737,"threads_started":1}
I20260806 02:52:57.206636 5918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "999e47b3a8ac4a18bd2eb72234307852" candidate_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "048e0815a5df436985507ef1fae29d80"
I20260806 02:52:57.206892 5918 raft_consensus.cc:3037] T 999e47b3a8ac4a18bd2eb72234307852 P 048e0815a5df436985507ef1fae29d80 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:57.207748 5702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "999e47b3a8ac4a18bd2eb72234307852" candidate_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90c9940f7ea24833bb84133bf92d11df" is_pre_election: true
I20260806 02:52:57.207788 5701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "999e47b3a8ac4a18bd2eb72234307852" candidate_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90c9940f7ea24833bb84133bf92d11df"
I20260806 02:52:57.208032 5702 raft_consensus.cc:2454] T 999e47b3a8ac4a18bd2eb72234307852 P 90c9940f7ea24833bb84133bf92d11df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab9b2267b3dc4fc794f9a3f59345fc80 in term 0.
I20260806 02:52:57.209121 6050 leader_election.cc:288] T 999e47b3a8ac4a18bd2eb72234307852 P ab9b2267b3dc4fc794f9a3f59345fc80 [CANDIDATE]: Term 1 election: Requested vote from peers 90c9940f7ea24833bb84133bf92d11df (127.25.211.130:40117), 048e0815a5df436985507ef1fae29d80 (127.25.211.131:38389)
I20260806 02:52:57.209878 5918 raft_consensus.cc:2454] T 999e47b3a8ac4a18bd2eb72234307852 P 048e0815a5df436985507ef1fae29d80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab9b2267b3dc4fc794f9a3f59345fc80 in term 1.
I20260806 02:52:57.210342 5386 leader_election.cc:302] T 999e47b3a8ac4a18bd2eb72234307852 P ab9b2267b3dc4fc794f9a3f59345fc80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 048e0815a5df436985507ef1fae29d80, ab9b2267b3dc4fc794f9a3f59345fc80; no voters: 90c9940f7ea24833bb84133bf92d11df
I20260806 02:52:57.210642 6050 raft_consensus.cc:2781] T 999e47b3a8ac4a18bd2eb72234307852 P ab9b2267b3dc4fc794f9a3f59345fc80 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:52:57.212953 6050 raft_consensus.cc:686] T 999e47b3a8ac4a18bd2eb72234307852 P ab9b2267b3dc4fc794f9a3f59345fc80 [term 1 LEADER]: Becoming Leader. State: Replica: ab9b2267b3dc4fc794f9a3f59345fc80, State: Running, Role: LEADER
I20260806 02:52:57.213411 6050 consensus_queue.cc:237] T 999e47b3a8ac4a18bd2eb72234307852 P ab9b2267b3dc4fc794f9a3f59345fc80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42013 } } peers { permanent_uuid: "90c9940f7ea24833bb84133bf92d11df" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40117 } } peers { permanent_uuid: "048e0815a5df436985507ef1fae29d80" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38389 } }
I20260806 02:52:57.216951 5253 catalog_manager.cc:5495] T 999e47b3a8ac4a18bd2eb72234307852 P ab9b2267b3dc4fc794f9a3f59345fc80 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab9b2267b3dc4fc794f9a3f59345fc80 (127.25.211.129). New cstate: current_term: 1 leader_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90c9940f7ea24833bb84133bf92d11df" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "048e0815a5df436985507ef1fae29d80" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38389 } health_report { overall_health: UNKNOWN } } }
I20260806 02:52:57.296288 6164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:57.289268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34430 (local address 127.25.211.129:42013)
0806 02:52:57.289854 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:57.289861 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:57.289884 (+ 23us) server_negotiation.cc:407] Connection header received
0806 02:52:57.290021 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:57.290027 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:57.290199 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:57.290413 (+ 214us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:52:57.290804 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:57.291948 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:57.293318 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:57.293958 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:57.294097 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:57.294745 (+ 648us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:52:57.295575 (+ 830us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:52:57.295739 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:57.295796 (+ 57us) server_negotiation.cc:299] Negotiation successful
0806 02:52:57.295889 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":333,"threads_started":1}
I20260806 02:52:57.301014 5701 raft_consensus.cc:3037] T 999e47b3a8ac4a18bd2eb72234307852 P 90c9940f7ea24833bb84133bf92d11df [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:57.301076 5918 raft_consensus.cc:1261] T 999e47b3a8ac4a18bd2eb72234307852 P 048e0815a5df436985507ef1fae29d80 [term 1 FOLLOWER]: Refusing update from remote peer ab9b2267b3dc4fc794f9a3f59345fc80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:52:57.301810 6158 consensus_queue.cc:1035] T 999e47b3a8ac4a18bd2eb72234307852 P ab9b2267b3dc4fc794f9a3f59345fc80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "048e0815a5df436985507ef1fae29d80" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:52:57.304661 5701 raft_consensus.cc:1261] T 999e47b3a8ac4a18bd2eb72234307852 P 90c9940f7ea24833bb84133bf92d11df [term 1 FOLLOWER]: Refusing update from remote peer ab9b2267b3dc4fc794f9a3f59345fc80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:52:57.305549 6158 consensus_queue.cc:1035] T 999e47b3a8ac4a18bd2eb72234307852 P ab9b2267b3dc4fc794f9a3f59345fc80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90c9940f7ea24833bb84133bf92d11df" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:52:57.345712 5254 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:52:57.345958 5254 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:52:57.349112 5254 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8c91b83144bf42b38480be74a574bb17: Sending DeleteTablet for 3 replicas of tablet 999e47b3a8ac4a18bd2eb72234307852
I20260806 02:52:57.350067 5893 tablet_service.cc:1514] Processing DeleteTablet for tablet 999e47b3a8ac4a18bd2eb72234307852 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:52:57 UTC) from {username='oryx', principal='oryx/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:37790
I20260806 02:52:57.350096 5458 tablet_service.cc:1514] Processing DeleteTablet for tablet 999e47b3a8ac4a18bd2eb72234307852 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:52:57 UTC) from {username='oryx', principal='oryx/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:34414
I20260806 02:52:57.351101 5671 tablet_service.cc:1514] Processing DeleteTablet for tablet 999e47b3a8ac4a18bd2eb72234307852 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:52:57 UTC) from {username='oryx', principal='oryx/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:59784
I20260806 02:52:57.359548 5254 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56628:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:52:57.360585 5254 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:52:57.365192 6185 tablet_replica.cc:330] T 999e47b3a8ac4a18bd2eb72234307852 P 90c9940f7ea24833bb84133bf92d11df: stopping tablet replica
I20260806 02:52:57.366099 6185 raft_consensus.cc:2229] T 999e47b3a8ac4a18bd2eb72234307852 P 90c9940f7ea24833bb84133bf92d11df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:52:57.371762 6185 raft_consensus.cc:2258] T 999e47b3a8ac4a18bd2eb72234307852 P 90c9940f7ea24833bb84133bf92d11df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:52:57.371932 5891 tablet_service.cc:1467] Processing CreateTablet for tablet 1482f21c9a1a4914bad9833d869b55f1 (DEFAULT_TABLE table=test-table [id=d51b88555ef247c38dc9de9d5d0d232e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:52:57.371946 5456 tablet_service.cc:1467] Processing CreateTablet for tablet 1482f21c9a1a4914bad9833d869b55f1 (DEFAULT_TABLE table=test-table [id=d51b88555ef247c38dc9de9d5d0d232e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:52:57.372437 5891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1482f21c9a1a4914bad9833d869b55f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:57.372565 5456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1482f21c9a1a4914bad9833d869b55f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:57.375056 6185 ts_tablet_manager.cc:1905] T 999e47b3a8ac4a18bd2eb72234307852 P 90c9940f7ea24833bb84133bf92d11df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:52:57.379216 6041 tablet_bootstrap.cc:492] T 1482f21c9a1a4914bad9833d869b55f1 P ab9b2267b3dc4fc794f9a3f59345fc80: Bootstrap starting.
I20260806 02:52:57.379253 5671 tablet_service.cc:1467] Processing CreateTablet for tablet 1482f21c9a1a4914bad9833d869b55f1 (DEFAULT_TABLE table=test-table [id=d51b88555ef247c38dc9de9d5d0d232e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:52:57.380532 5671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1482f21c9a1a4914bad9833d869b55f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:52:57.380923 6041 tablet_bootstrap.cc:654] T 1482f21c9a1a4914bad9833d869b55f1 P ab9b2267b3dc4fc794f9a3f59345fc80: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:57.381201 6040 tablet_bootstrap.cc:492] T 1482f21c9a1a4914bad9833d869b55f1 P 048e0815a5df436985507ef1fae29d80: Bootstrap starting.
I20260806 02:52:57.382931 6040 tablet_bootstrap.cc:654] T 1482f21c9a1a4914bad9833d869b55f1 P 048e0815a5df436985507ef1fae29d80: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:57.389587 6040 tablet_bootstrap.cc:492] T 1482f21c9a1a4914bad9833d869b55f1 P 048e0815a5df436985507ef1fae29d80: No bootstrap required, opened a new log
I20260806 02:52:57.389868 6040 ts_tablet_manager.cc:1397] T 1482f21c9a1a4914bad9833d869b55f1 P 048e0815a5df436985507ef1fae29d80: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.001s
I20260806 02:52:57.390928 6040 raft_consensus.cc:348] T 1482f21c9a1a4914bad9833d869b55f1 P 048e0815a5df436985507ef1fae29d80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42013 } } peers { permanent_uuid: "90c9940f7ea24833bb84133bf92d11df" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40117 } } peers { permanent_uuid: "048e0815a5df436985507ef1fae29d80" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38389 } }
I20260806 02:52:57.391330 6040 raft_consensus.cc:374] T 1482f21c9a1a4914bad9833d869b55f1 P 048e0815a5df436985507ef1fae29d80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:57.391458 6040 raft_consensus.cc:729] T 1482f21c9a1a4914bad9833d869b55f1 P 048e0815a5df436985507ef1fae29d80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 048e0815a5df436985507ef1fae29d80, State: Initialized, Role: FOLLOWER
I20260806 02:52:57.391803 6040 consensus_queue.cc:260] T 1482f21c9a1a4914bad9833d869b55f1 P 048e0815a5df436985507ef1fae29d80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42013 } } peers { permanent_uuid: "90c9940f7ea24833bb84133bf92d11df" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40117 } } peers { permanent_uuid: "048e0815a5df436985507ef1fae29d80" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38389 } }
I20260806 02:52:57.393148 6040 ts_tablet_manager.cc:1428] T 1482f21c9a1a4914bad9833d869b55f1 P 048e0815a5df436985507ef1fae29d80: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260806 02:52:57.399096 6182 tablet_replica.cc:330] T 999e47b3a8ac4a18bd2eb72234307852 P 048e0815a5df436985507ef1fae29d80: stopping tablet replica
I20260806 02:52:57.399528 6039 tablet_bootstrap.cc:492] T 1482f21c9a1a4914bad9833d869b55f1 P 90c9940f7ea24833bb84133bf92d11df: Bootstrap starting.
I20260806 02:52:57.401453 6039 tablet_bootstrap.cc:654] T 1482f21c9a1a4914bad9833d869b55f1 P 90c9940f7ea24833bb84133bf92d11df: Neither blocks nor log segments found. Creating new log.
I20260806 02:52:57.401811 6041 tablet_bootstrap.cc:492] T 1482f21c9a1a4914bad9833d869b55f1 P ab9b2267b3dc4fc794f9a3f59345fc80: No bootstrap required, opened a new log
I20260806 02:52:57.402070 6041 ts_tablet_manager.cc:1397] T 1482f21c9a1a4914bad9833d869b55f1 P ab9b2267b3dc4fc794f9a3f59345fc80: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.013s
I20260806 02:52:57.403056 6041 raft_consensus.cc:348] T 1482f21c9a1a4914bad9833d869b55f1 P ab9b2267b3dc4fc794f9a3f59345fc80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42013 } } peers { permanent_uuid: "90c9940f7ea24833bb84133bf92d11df" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40117 } } peers { permanent_uuid: "048e0815a5df436985507ef1fae29d80" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38389 } }
I20260806 02:52:57.403466 6041 raft_consensus.cc:374] T 1482f21c9a1a4914bad9833d869b55f1 P ab9b2267b3dc4fc794f9a3f59345fc80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:57.403605 6041 raft_consensus.cc:729] T 1482f21c9a1a4914bad9833d869b55f1 P ab9b2267b3dc4fc794f9a3f59345fc80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab9b2267b3dc4fc794f9a3f59345fc80, State: Initialized, Role: FOLLOWER
I20260806 02:52:57.403971 6041 consensus_queue.cc:260] T 1482f21c9a1a4914bad9833d869b55f1 P ab9b2267b3dc4fc794f9a3f59345fc80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42013 } } peers { permanent_uuid: "90c9940f7ea24833bb84133bf92d11df" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40117 } } peers { permanent_uuid: "048e0815a5df436985507ef1fae29d80" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38389 } }
I20260806 02:52:57.406070 6182 raft_consensus.cc:2229] T 999e47b3a8ac4a18bd2eb72234307852 P 048e0815a5df436985507ef1fae29d80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:52:57.406468 6182 raft_consensus.cc:2258] T 999e47b3a8ac4a18bd2eb72234307852 P 048e0815a5df436985507ef1fae29d80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:52:57.406654 6183 tablet_replica.cc:330] T 999e47b3a8ac4a18bd2eb72234307852 P ab9b2267b3dc4fc794f9a3f59345fc80: stopping tablet replica
I20260806 02:52:57.406877 6183 raft_consensus.cc:2229] T 999e47b3a8ac4a18bd2eb72234307852 P ab9b2267b3dc4fc794f9a3f59345fc80 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:52:57.407984 6182 ts_tablet_manager.cc:1905] T 999e47b3a8ac4a18bd2eb72234307852 P 048e0815a5df436985507ef1fae29d80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:52:57.408157 6039 tablet_bootstrap.cc:492] T 1482f21c9a1a4914bad9833d869b55f1 P 90c9940f7ea24833bb84133bf92d11df: No bootstrap required, opened a new log
I20260806 02:52:57.407801 6183 raft_consensus.cc:2258] T 999e47b3a8ac4a18bd2eb72234307852 P ab9b2267b3dc4fc794f9a3f59345fc80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:52:57.408335 6039 ts_tablet_manager.cc:1397] T 1482f21c9a1a4914bad9833d869b55f1 P 90c9940f7ea24833bb84133bf92d11df: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20260806 02:52:57.409521 6041 ts_tablet_manager.cc:1428] T 1482f21c9a1a4914bad9833d869b55f1 P ab9b2267b3dc4fc794f9a3f59345fc80: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20260806 02:52:57.409453 6039 raft_consensus.cc:348] T 1482f21c9a1a4914bad9833d869b55f1 P 90c9940f7ea24833bb84133bf92d11df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42013 } } peers { permanent_uuid: "90c9940f7ea24833bb84133bf92d11df" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40117 } } peers { permanent_uuid: "048e0815a5df436985507ef1fae29d80" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38389 } }
I20260806 02:52:57.409740 6039 raft_consensus.cc:374] T 1482f21c9a1a4914bad9833d869b55f1 P 90c9940f7ea24833bb84133bf92d11df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:52:57.409828 6039 raft_consensus.cc:729] T 1482f21c9a1a4914bad9833d869b55f1 P 90c9940f7ea24833bb84133bf92d11df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90c9940f7ea24833bb84133bf92d11df, State: Initialized, Role: FOLLOWER
I20260806 02:52:57.410113 6039 consensus_queue.cc:260] T 1482f21c9a1a4914bad9833d869b55f1 P 90c9940f7ea24833bb84133bf92d11df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42013 } } peers { permanent_uuid: "90c9940f7ea24833bb84133bf92d11df" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40117 } } peers { permanent_uuid: "048e0815a5df436985507ef1fae29d80" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38389 } }
I20260806 02:52:57.411288 6039 ts_tablet_manager.cc:1428] T 1482f21c9a1a4914bad9833d869b55f1 P 90c9940f7ea24833bb84133bf92d11df: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260806 02:52:57.415201 6183 ts_tablet_manager.cc:1905] T 999e47b3a8ac4a18bd2eb72234307852 P ab9b2267b3dc4fc794f9a3f59345fc80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:52:57.406114 6185 ts_tablet_manager.cc:1918] T 999e47b3a8ac4a18bd2eb72234307852 P 90c9940f7ea24833bb84133bf92d11df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:52:57.418205 6182 ts_tablet_manager.cc:1918] T 999e47b3a8ac4a18bd2eb72234307852 P 048e0815a5df436985507ef1fae29d80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:52:57.418351 6182 log.cc:1196] T 999e47b3a8ac4a18bd2eb72234307852 P 048e0815a5df436985507ef1fae29d80: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/ts-2/wal/wals/999e47b3a8ac4a18bd2eb72234307852
I20260806 02:52:57.418359 6185 log.cc:1196] T 999e47b3a8ac4a18bd2eb72234307852 P 90c9940f7ea24833bb84133bf92d11df: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/ts-1/wal/wals/999e47b3a8ac4a18bd2eb72234307852
I20260806 02:52:57.418841 6185 ts_tablet_manager.cc:1939] T 999e47b3a8ac4a18bd2eb72234307852 P 90c9940f7ea24833bb84133bf92d11df: Deleting consensus metadata
I20260806 02:52:57.418848 6182 ts_tablet_manager.cc:1939] T 999e47b3a8ac4a18bd2eb72234307852 P 048e0815a5df436985507ef1fae29d80: Deleting consensus metadata
I20260806 02:52:57.420202 5234 catalog_manager.cc:4841] TS 90c9940f7ea24833bb84133bf92d11df (127.25.211.130:40117): tablet 999e47b3a8ac4a18bd2eb72234307852 (table test-table [id=4d76c4a9408440fdbf4d212b0ada5864]) successfully deleted
I20260806 02:52:57.420531 5238 catalog_manager.cc:4841] TS 048e0815a5df436985507ef1fae29d80 (127.25.211.131:38389): tablet 999e47b3a8ac4a18bd2eb72234307852 (table test-table [id=4d76c4a9408440fdbf4d212b0ada5864]) successfully deleted
I20260806 02:52:57.424628 6183 ts_tablet_manager.cc:1918] T 999e47b3a8ac4a18bd2eb72234307852 P ab9b2267b3dc4fc794f9a3f59345fc80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:52:57.424814 6183 log.cc:1196] T 999e47b3a8ac4a18bd2eb72234307852 P ab9b2267b3dc4fc794f9a3f59345fc80: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/ts-0/wal/wals/999e47b3a8ac4a18bd2eb72234307852
I20260806 02:52:57.425338 6183 ts_tablet_manager.cc:1939] T 999e47b3a8ac4a18bd2eb72234307852 P ab9b2267b3dc4fc794f9a3f59345fc80: Deleting consensus metadata
I20260806 02:52:57.426816 5235 catalog_manager.cc:4841] TS ab9b2267b3dc4fc794f9a3f59345fc80 (127.25.211.129:42013): tablet 999e47b3a8ac4a18bd2eb72234307852 (table test-table [id=4d76c4a9408440fdbf4d212b0ada5864]) successfully deleted
I20260806 02:52:57.582919 6050 raft_consensus.cc:482] T 1482f21c9a1a4914bad9833d869b55f1 P ab9b2267b3dc4fc794f9a3f59345fc80 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:52:57.583189 6050 raft_consensus.cc:504] T 1482f21c9a1a4914bad9833d869b55f1 P ab9b2267b3dc4fc794f9a3f59345fc80 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42013 } } peers { permanent_uuid: "90c9940f7ea24833bb84133bf92d11df" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40117 } } peers { permanent_uuid: "048e0815a5df436985507ef1fae29d80" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38389 } }
I20260806 02:52:57.583899 6050 leader_election.cc:288] T 1482f21c9a1a4914bad9833d869b55f1 P ab9b2267b3dc4fc794f9a3f59345fc80 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90c9940f7ea24833bb84133bf92d11df (127.25.211.130:40117), 048e0815a5df436985507ef1fae29d80 (127.25.211.131:38389)
I20260806 02:52:57.584630 5918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1482f21c9a1a4914bad9833d869b55f1" candidate_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "048e0815a5df436985507ef1fae29d80" is_pre_election: true
I20260806 02:52:57.584652 5701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1482f21c9a1a4914bad9833d869b55f1" candidate_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90c9940f7ea24833bb84133bf92d11df" is_pre_election: true
I20260806 02:52:57.584936 5918 raft_consensus.cc:2454] T 1482f21c9a1a4914bad9833d869b55f1 P 048e0815a5df436985507ef1fae29d80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab9b2267b3dc4fc794f9a3f59345fc80 in term 0.
I20260806 02:52:57.584960 5701 raft_consensus.cc:2454] T 1482f21c9a1a4914bad9833d869b55f1 P 90c9940f7ea24833bb84133bf92d11df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab9b2267b3dc4fc794f9a3f59345fc80 in term 0.
I20260806 02:52:57.585475 5386 leader_election.cc:302] T 1482f21c9a1a4914bad9833d869b55f1 P ab9b2267b3dc4fc794f9a3f59345fc80 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 048e0815a5df436985507ef1fae29d80, 90c9940f7ea24833bb84133bf92d11df, ab9b2267b3dc4fc794f9a3f59345fc80; no voters:
I20260806 02:52:57.585922 6050 raft_consensus.cc:2781] T 1482f21c9a1a4914bad9833d869b55f1 P ab9b2267b3dc4fc794f9a3f59345fc80 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:52:57.586022 6050 raft_consensus.cc:482] T 1482f21c9a1a4914bad9833d869b55f1 P ab9b2267b3dc4fc794f9a3f59345fc80 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:52:57.586128 6050 raft_consensus.cc:3037] T 1482f21c9a1a4914bad9833d869b55f1 P ab9b2267b3dc4fc794f9a3f59345fc80 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:57.588485 6050 raft_consensus.cc:504] T 1482f21c9a1a4914bad9833d869b55f1 P ab9b2267b3dc4fc794f9a3f59345fc80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42013 } } peers { permanent_uuid: "90c9940f7ea24833bb84133bf92d11df" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40117 } } peers { permanent_uuid: "048e0815a5df436985507ef1fae29d80" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38389 } }
I20260806 02:52:57.589399 6050 leader_election.cc:288] T 1482f21c9a1a4914bad9833d869b55f1 P ab9b2267b3dc4fc794f9a3f59345fc80 [CANDIDATE]: Term 1 election: Requested vote from peers 90c9940f7ea24833bb84133bf92d11df (127.25.211.130:40117), 048e0815a5df436985507ef1fae29d80 (127.25.211.131:38389)
I20260806 02:52:57.589906 5701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1482f21c9a1a4914bad9833d869b55f1" candidate_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90c9940f7ea24833bb84133bf92d11df"
I20260806 02:52:57.589948 5918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1482f21c9a1a4914bad9833d869b55f1" candidate_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "048e0815a5df436985507ef1fae29d80"
I20260806 02:52:57.590205 5918 raft_consensus.cc:3037] T 1482f21c9a1a4914bad9833d869b55f1 P 048e0815a5df436985507ef1fae29d80 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:57.590224 5701 raft_consensus.cc:3037] T 1482f21c9a1a4914bad9833d869b55f1 P 90c9940f7ea24833bb84133bf92d11df [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:52:57.593428 5701 raft_consensus.cc:2454] T 1482f21c9a1a4914bad9833d869b55f1 P 90c9940f7ea24833bb84133bf92d11df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab9b2267b3dc4fc794f9a3f59345fc80 in term 1.
I20260806 02:52:57.593444 5918 raft_consensus.cc:2454] T 1482f21c9a1a4914bad9833d869b55f1 P 048e0815a5df436985507ef1fae29d80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab9b2267b3dc4fc794f9a3f59345fc80 in term 1.
I20260806 02:52:57.594029 5380 leader_election.cc:302] T 1482f21c9a1a4914bad9833d869b55f1 P ab9b2267b3dc4fc794f9a3f59345fc80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90c9940f7ea24833bb84133bf92d11df, ab9b2267b3dc4fc794f9a3f59345fc80; no voters:
I20260806 02:52:57.594413 6050 raft_consensus.cc:2781] T 1482f21c9a1a4914bad9833d869b55f1 P ab9b2267b3dc4fc794f9a3f59345fc80 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:52:57.594645 6050 raft_consensus.cc:686] T 1482f21c9a1a4914bad9833d869b55f1 P ab9b2267b3dc4fc794f9a3f59345fc80 [term 1 LEADER]: Becoming Leader. State: Replica: ab9b2267b3dc4fc794f9a3f59345fc80, State: Running, Role: LEADER
I20260806 02:52:57.595079 6050 consensus_queue.cc:237] T 1482f21c9a1a4914bad9833d869b55f1 P ab9b2267b3dc4fc794f9a3f59345fc80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42013 } } peers { permanent_uuid: "90c9940f7ea24833bb84133bf92d11df" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40117 } } peers { permanent_uuid: "048e0815a5df436985507ef1fae29d80" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38389 } }
I20260806 02:52:57.598390 5253 catalog_manager.cc:5495] T 1482f21c9a1a4914bad9833d869b55f1 P ab9b2267b3dc4fc794f9a3f59345fc80 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab9b2267b3dc4fc794f9a3f59345fc80 (127.25.211.129). New cstate: current_term: 1 leader_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab9b2267b3dc4fc794f9a3f59345fc80" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90c9940f7ea24833bb84133bf92d11df" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "048e0815a5df436985507ef1fae29d80" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38389 } health_report { overall_health: UNKNOWN } } }
I20260806 02:52:57.661139 6197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:57.651362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.130:40117 (local address 127.25.211.129:52857)
0806 02:52:57.654798 (+ 3436us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:57.654821 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:57.655074 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:57.655521 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:57.655527 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:57.655544 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:52:57.655948 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:57.655961 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:57.657484 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:57.657488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:57.658844 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:57.658864 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:57.659923 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:57.659927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:57.660054 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:57.660131 (+ 77us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:52:57.660601 (+ 470us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:52:57.660610 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 02:52:57.660677 (+ 67us) client_negotiation.cc:240] Negotiation successful
0806 02:52:57.660803 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3215,"thread_start_us":3004,"threads_started":1}
I20260806 02:52:57.661119 6156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:57.651352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:52857 (local address 127.25.211.130:40117)
0806 02:52:57.651568 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:57.651575 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:57.655026 (+ 3451us) server_negotiation.cc:407] Connection header received
0806 02:52:57.655258 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:57.655265 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:57.655363 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:57.655473 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:52:57.656198 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:57.657301 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:57.659035 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:57.659713 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:57.659864 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:57.660342 (+ 478us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:52:57.660492 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:52:57.660592 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:57.660727 (+ 135us) server_negotiation.cc:299] Negotiation successful
0806 02:52:57.660822 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 02:52:57.671484 5918 raft_consensus.cc:1261] T 1482f21c9a1a4914bad9833d869b55f1 P 048e0815a5df436985507ef1fae29d80 [term 1 FOLLOWER]: Refusing update from remote peer ab9b2267b3dc4fc794f9a3f59345fc80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:52:57.671543 5701 raft_consensus.cc:1261] T 1482f21c9a1a4914bad9833d869b55f1 P 90c9940f7ea24833bb84133bf92d11df [term 1 FOLLOWER]: Refusing update from remote peer ab9b2267b3dc4fc794f9a3f59345fc80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:52:57.672299 6050 consensus_queue.cc:1035] T 1482f21c9a1a4914bad9833d869b55f1 P ab9b2267b3dc4fc794f9a3f59345fc80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "048e0815a5df436985507ef1fae29d80" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:52:57.672590 6158 consensus_queue.cc:1035] T 1482f21c9a1a4914bad9833d869b55f1 P ab9b2267b3dc4fc794f9a3f59345fc80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90c9940f7ea24833bb84133bf92d11df" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:52:57.722249 6164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:57.715262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:56225 (local address 127.25.211.129:42013)
0806 02:52:57.715420 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:57.715424 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:57.715967 (+ 543us) server_negotiation.cc:407] Connection header received
0806 02:52:57.716284 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:57.716290 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:57.716389 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:57.716507 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:52:57.717184 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:57.718385 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:57.720460 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:57.720919 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:57.721024 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:57.721550 (+ 526us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:52:57.721653 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:52:57.721719 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:57.721895 (+ 176us) server_negotiation.cc:299] Negotiation successful
0806 02:52:57.721963 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260806 02:52:57.722466 6205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:57.715237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:42013 (local address 127.25.211.130:56225)
0806 02:52:57.715865 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0806 02:52:57.715885 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 02:52:57.716089 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:52:57.716539 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:52:57.716544 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:52:57.716557 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 02:52:57.716979 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:57.716993 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:57.718597 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:57.718603 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:57.720293 (+ 1690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:52:57.720312 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:52:57.721125 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:52:57.721132 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:52:57.721329 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:57.721385 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 02:52:57.721773 (+ 388us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 02:52:57.721786 (+ 13us) client_negotiation.cc:769] Sending connection context
0806 02:52:57.721876 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 02:52:57.722036 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":1307,"threads_started":1}
I20260806 02:52:57.768327 5253 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:52:57.768710 5253 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:52:57.772764 5253 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8c91b83144bf42b38480be74a574bb17: Sending DeleteTablet for 3 replicas of tablet 1482f21c9a1a4914bad9833d869b55f1
I20260806 02:52:57.773798 5458 tablet_service.cc:1514] Processing DeleteTablet for tablet 1482f21c9a1a4914bad9833d869b55f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:52:57 UTC) from {username='oryx', principal='oryx/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:34414
I20260806 02:52:57.773909 5893 tablet_service.cc:1514] Processing DeleteTablet for tablet 1482f21c9a1a4914bad9833d869b55f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:52:57 UTC) from {username='oryx', principal='oryx/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:37790
I20260806 02:52:57.774246 6183 tablet_replica.cc:330] T 1482f21c9a1a4914bad9833d869b55f1 P ab9b2267b3dc4fc794f9a3f59345fc80: stopping tablet replica
I20260806 02:52:57.774446 6183 raft_consensus.cc:2229] T 1482f21c9a1a4914bad9833d869b55f1 P ab9b2267b3dc4fc794f9a3f59345fc80 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:52:57.774816 6183 raft_consensus.cc:2258] T 1482f21c9a1a4914bad9833d869b55f1 P ab9b2267b3dc4fc794f9a3f59345fc80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:52:57.775147 6182 tablet_replica.cc:330] T 1482f21c9a1a4914bad9833d869b55f1 P 048e0815a5df436985507ef1fae29d80: stopping tablet replica
I20260806 02:52:57.775353 6182 raft_consensus.cc:2229] T 1482f21c9a1a4914bad9833d869b55f1 P 048e0815a5df436985507ef1fae29d80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:52:57.775540 6182 raft_consensus.cc:2258] T 1482f21c9a1a4914bad9833d869b55f1 P 048e0815a5df436985507ef1fae29d80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:52:57.776741 6183 ts_tablet_manager.cc:1905] T 1482f21c9a1a4914bad9833d869b55f1 P ab9b2267b3dc4fc794f9a3f59345fc80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:52:57.778851 6182 ts_tablet_manager.cc:1905] T 1482f21c9a1a4914bad9833d869b55f1 P 048e0815a5df436985507ef1fae29d80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:52:57.784230 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 5357
I20260806 02:52:57.784276 6183 ts_tablet_manager.cc:1918] T 1482f21c9a1a4914bad9833d869b55f1 P ab9b2267b3dc4fc794f9a3f59345fc80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:52:57.784467 6183 log.cc:1196] T 1482f21c9a1a4914bad9833d869b55f1 P ab9b2267b3dc4fc794f9a3f59345fc80: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/ts-0/wal/wals/1482f21c9a1a4914bad9833d869b55f1
I20260806 02:52:57.785732 6182 ts_tablet_manager.cc:1918] T 1482f21c9a1a4914bad9833d869b55f1 P 048e0815a5df436985507ef1fae29d80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:52:57.785900 6182 log.cc:1196] T 1482f21c9a1a4914bad9833d869b55f1 P 048e0815a5df436985507ef1fae29d80: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/ts-2/wal/wals/1482f21c9a1a4914bad9833d869b55f1
I20260806 02:52:57.786363 6182 ts_tablet_manager.cc:1939] T 1482f21c9a1a4914bad9833d869b55f1 P 048e0815a5df436985507ef1fae29d80: Deleting consensus metadata
I20260806 02:52:57.787523 5238 catalog_manager.cc:4841] TS 048e0815a5df436985507ef1fae29d80 (127.25.211.131:38389): tablet 1482f21c9a1a4914bad9833d869b55f1 (table test-table [id=d51b88555ef247c38dc9de9d5d0d232e]) successfully deleted
I20260806 02:52:57.787735 5671 tablet_service.cc:1514] Processing DeleteTablet for tablet 1482f21c9a1a4914bad9833d869b55f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:52:57 UTC) from {username='oryx', principal='oryx/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:59784
I20260806 02:52:57.788059 6185 tablet_replica.cc:330] T 1482f21c9a1a4914bad9833d869b55f1 P 90c9940f7ea24833bb84133bf92d11df: stopping tablet replica
I20260806 02:52:57.788182 6183 ts_tablet_manager.cc:1939] T 1482f21c9a1a4914bad9833d869b55f1 P ab9b2267b3dc4fc794f9a3f59345fc80: Deleting consensus metadata
I20260806 02:52:57.788266 6185 raft_consensus.cc:2229] T 1482f21c9a1a4914bad9833d869b55f1 P 90c9940f7ea24833bb84133bf92d11df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:52:57.788481 6185 raft_consensus.cc:2258] T 1482f21c9a1a4914bad9833d869b55f1 P 90c9940f7ea24833bb84133bf92d11df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:52:57.789425 5235 catalog_manager.cc:4841] TS ab9b2267b3dc4fc794f9a3f59345fc80 (127.25.211.129:42013): tablet 1482f21c9a1a4914bad9833d869b55f1 (table test-table [id=d51b88555ef247c38dc9de9d5d0d232e]) successfully deleted
I20260806 02:52:57.789855 6185 ts_tablet_manager.cc:1905] T 1482f21c9a1a4914bad9833d869b55f1 P 90c9940f7ea24833bb84133bf92d11df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:52:57.797207 6185 ts_tablet_manager.cc:1918] T 1482f21c9a1a4914bad9833d869b55f1 P 90c9940f7ea24833bb84133bf92d11df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 02:52:57.797441 6185 log.cc:1196] T 1482f21c9a1a4914bad9833d869b55f1 P 90c9940f7ea24833bb84133bf92d11df: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1785984707115702-26446-0/minicluster-data/ts-1/wal/wals/1482f21c9a1a4914bad9833d869b55f1
I20260806 02:52:57.798012 6185 ts_tablet_manager.cc:1939] T 1482f21c9a1a4914bad9833d869b55f1 P 90c9940f7ea24833bb84133bf92d11df: Deleting consensus metadata
Aug 06 02:52:57 dist-test-slave-zj8q krb5kdc[5196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.25.211.129@KRBTEST.COM
I20260806 02:52:57.799429 5234 catalog_manager.cc:4841] TS 90c9940f7ea24833bb84133bf92d11df (127.25.211.130:40117): tablet 1482f21c9a1a4914bad9833d869b55f1 (table test-table [id=d51b88555ef247c38dc9de9d5d0d232e]) successfully deleted
I20260806 02:52:57.808147 6164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:57.785748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34446 (local address 127.25.211.129:42013)
0806 02:52:57.785944 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:57.785950 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:57.787934 (+ 1984us) server_negotiation.cc:407] Connection header received
0806 02:52:57.788262 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:57.788268 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:57.788355 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:57.788457 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:57.790622 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:57.791782 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:57.793627 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:57.794379 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:57.794478 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:57.799880 (+ 5402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:57.799903 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:57.799928 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:57.799986 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:57.804355 (+ 4369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:57.805353 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:57.805363 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:57.805368 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:57.805563 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:57.806040 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:57.806046 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:57.806050 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:57.806632 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:57.806961 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:57.807440 (+ 479us) server_negotiation.cc:299] Negotiation successful
0806 02:52:57.807707 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 02:52:57.808871 5552 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:52:57.952306 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 5357
I20260806 02:52:57.979296 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 5563
Aug 06 02:52:57 dist-test-slave-zj8q krb5kdc[5196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.25.211.130@KRBTEST.COM
I20260806 02:52:57.995369 6156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:57.980288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59808 (local address 127.25.211.130:40117)
0806 02:52:57.980498 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:57.980504 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:57.980526 (+ 22us) server_negotiation.cc:407] Connection header received
0806 02:52:57.980603 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:57.980611 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:57.980679 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:57.980772 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:57.982184 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:57.982954 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:57.984328 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:57.984792 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:57.984912 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:57.989703 (+ 4791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:57.989723 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:57.989741 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:57.989783 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:57.992611 (+ 2828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:57.993332 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:57.993340 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:57.993342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:57.993478 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:57.993791 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:57.993794 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:57.993796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:57.994284 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:57.994556 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:57.994884 (+ 328us) server_negotiation.cc:299] Negotiation successful
0806 02:52:57.995097 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 02:52:57.996157 5769 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:52:58.146003 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 5563
I20260806 02:52:58.177397 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 5780
Aug 06 02:52:58 dist-test-slave-zj8q krb5kdc[5196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.25.211.131@KRBTEST.COM
I20260806 02:52:58.193714 6215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:52:58.178271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37804 (local address 127.25.211.131:38389)
0806 02:52:58.178939 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0806 02:52:58.178944 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:52:58.178962 (+ 18us) server_negotiation.cc:407] Connection header received
0806 02:52:58.179020 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:52:58.179024 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:52:58.179086 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:52:58.179180 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:52:58.180576 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:58.181372 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:58.182796 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:52:58.183212 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:52:58.183318 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:52:58.187872 (+ 4554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:52:58.187888 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:52:58.187906 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:52:58.187957 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:52:58.190951 (+ 2994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:58.191700 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:58.191708 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:58.191711 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:58.191883 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:52:58.192246 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:52:58.192250 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:52:58.192252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:52:58.192632 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:52:58.192877 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0806 02:52:58.193119 (+ 242us) server_negotiation.cc:299] Negotiation successful
0806 02:52:58.193347 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":299,"threads_started":1}
I20260806 02:52:58.194422 5994 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:52:58.320253 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 5780
I20260806 02:52:58.349045 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 5216
I20260806 02:52:58.350361 5317 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:52:58.467567 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 5216
2026-08-06T02:52:58Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6517 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984707115702-26446-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:52:58 dist-test-slave-zj8q krb5kdc[6226](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:52:58 dist-test-slave-zj8q krb5kdc[6226](info): set up 2 sockets
Aug 06 02:52:58 dist-test-slave-zj8q krb5kdc[6226](info): commencing operation
krb5kdc: starting...
W20260806 02:52:59.702044 26446 mini_kdc.cc:121] Time spent starting KDC: real 1.187s user 0.002s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Aug 06 02:52:59 dist-test-slave-zj8q krb5kdc[6226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:52:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:52:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.211.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab.
Entry for principal kudu/127.25.211.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab.
I20260806 02:52:59.990326 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984707115702-26446-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984707115702-26446-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984707115702-26446-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:38501
--webserver_interface=127.25.211.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:42623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.211.190:38501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:53:00.346791 6246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:53:00.347121 6246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:53:00.347225 6246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:53:00.354799 6246 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:53:00.354933 6246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:53:00.354984 6246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:53:00.355070 6246 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:53:00.355135 6246 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:53:00.365557 6246 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:42623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984707115702-26446-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984707115702-26446-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.211.190:38501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:38501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.211.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1785984707115702-26446-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:53:00.367373 6246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:53:00.368782 6246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:53:00.381309 6253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:00.381311 6256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:00.381309 6254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:53:00.382081 6246 server_base.cc:1034] running on GCE node
I20260806 02:53:00.383003 6246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:53:00.384578 6246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:53:00.385838 6246 hybrid_clock.cc:648] HybridClock initialized: now 1785984780385771 us; error 45 us; skew 500 ppm
I20260806 02:53:00.386332 6246 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-06T02:53:00Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1996 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:53:00 dist-test-slave-zj8q krb5kdc[6263](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:53:00 dist-test-slave-zj8q krb5kdc[6263](info): set up 2 sockets
Aug 06 02:53:00 dist-test-slave-zj8q krb5kdc[6263](info): commencing operation
krb5kdc: starting...
W20260806 02:53:01.694911 26446 mini_kdc.cc:121] Time spent starting KDC: real 1.183s user 0.008s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Aug 06 02:53:01 dist-test-slave-zj8q krb5kdc[6263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:53:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:53:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.211.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab.
Entry for principal kudu/127.25.211.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab.
I20260806 02:53:01.998667 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:35411
--webserver_interface=127.25.211.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:38545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.211.190:35411
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:53:02.352703 6286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:53:02.353056 6286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:53:02.353153 6286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:53:02.360862 6286 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:53:02.360999 6286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:53:02.361047 6286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:53:02.361111 6286 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:53:02.361177 6286 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:53:02.372181 6286 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:38545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.211.190:35411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:35411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.211.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:53:02.373610 6286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:53:02.374997 6286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:53:02.387365 6299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:02.387421 6297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:02.387395 6296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:53:02.388170 6286 server_base.cc:1034] running on GCE node
I20260806 02:53:02.389051 6286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:53:02.390630 6286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:53:02.391867 6286 hybrid_clock.cc:648] HybridClock initialized: now 1785984782391803 us; error 52 us; skew 500 ppm
I20260806 02:53:02.392246 6286 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:53:02 dist-test-slave-zj8q krb5kdc[6263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984782, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:53:02.395721 6286 init.cc:377] Logged in from keytab as kudu/127.25.211.190@KRBTEST.COM (short username kudu)
I20260806 02:53:02.397367 6286 webserver.cc:460] Webserver started at http://127.25.211.190:42571/ using document root <none> and password file <none>
I20260806 02:53:02.398094 6286 fs_manager.cc:359] Metadata directory not provided
I20260806 02:53:02.398252 6286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:53:02.398576 6286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:53:02.401768 6286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/master-0/data/instance:
uuid: "4d569aec4a6f401c8804a0ba6ea3b79f"
format_stamp: "Formatted at 2026-08-06 02:53:02 on dist-test-slave-zj8q"
server_key: "0413d612db97780341f0aaffef5e8d25"
server_key_iv: "17602882ec43c7f5abaa658f3585978f"
server_key_version: "encryptionkey@0"
I20260806 02:53:02.402683 6286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/master-0/wal/instance:
uuid: "4d569aec4a6f401c8804a0ba6ea3b79f"
format_stamp: "Formatted at 2026-08-06 02:53:02 on dist-test-slave-zj8q"
server_key: "0413d612db97780341f0aaffef5e8d25"
server_key_iv: "17602882ec43c7f5abaa658f3585978f"
server_key_version: "encryptionkey@0"
I20260806 02:53:02.408703 6286 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 02:53:02.412662 6307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:02.414194 6286 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:53:02.414409 6286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/master-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/master-0/wal
uuid: "4d569aec4a6f401c8804a0ba6ea3b79f"
format_stamp: "Formatted at 2026-08-06 02:53:02 on dist-test-slave-zj8q"
server_key: "0413d612db97780341f0aaffef5e8d25"
server_key_iv: "17602882ec43c7f5abaa658f3585978f"
server_key_version: "encryptionkey@0"
I20260806 02:53:02.414619 6286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:53:02.427378 6286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:53:02.432250 6286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:53:02.432595 6286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:53:02.453032 6286 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.190:35411
I20260806 02:53:02.453156 6395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.190:35411 every 8 connection(s)
I20260806 02:53:02.454785 6286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
I20260806 02:53:02.459100 6396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:53:02.461777 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 6286
I20260806 02:53:02.461971 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/master-0/wal/instance
I20260806 02:53:02.462733 26446 external_mini_cluster.cc:1428] Setting key 2e39fc38f1bd52296bda80d5c574a70f
I20260806 02:53:02.471418 6396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d569aec4a6f401c8804a0ba6ea3b79f: Bootstrap starting.
I20260806 02:53:02.474853 6396 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d569aec4a6f401c8804a0ba6ea3b79f: Neither blocks nor log segments found. Creating new log.
I20260806 02:53:02.475956 6396 log.cc:824] T 00000000000000000000000000000000 P 4d569aec4a6f401c8804a0ba6ea3b79f: Log is configured to *not* fsync() on all Append() calls
Aug 06 02:53:02 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:53:02.479485 6396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d569aec4a6f401c8804a0ba6ea3b79f: No bootstrap required, opened a new log
I20260806 02:53:02.486641 6400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:02.465020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53616 (local address 127.25.211.190:35411)
0806 02:53:02.465911 (+ 891us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:02.465924 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:02.465969 (+ 45us) server_negotiation.cc:407] Connection header received
0806 02:53:02.467806 (+ 1837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:02.467833 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:02.468291 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:02.468689 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:02.470299 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:02.471675 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:02.473028 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:02.473765 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:02.473956 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:02.478237 (+ 4281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:02.478248 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:02.478266 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:53:02.478305 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:02.481774 (+ 3469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:02.482728 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:02.482736 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:02.482744 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:02.482948 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:02.483355 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:02.483359 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:02.483362 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:02.483969 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:02.484317 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:02.484875 (+ 558us) server_negotiation.cc:299] Negotiation successful
0806 02:53:02.485369 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":402,"threads_started":1}
I20260806 02:53:02.488265 6396 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d569aec4a6f401c8804a0ba6ea3b79f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d569aec4a6f401c8804a0ba6ea3b79f" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 35411 } }
I20260806 02:53:02.488693 6396 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d569aec4a6f401c8804a0ba6ea3b79f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:53:02.488802 6396 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d569aec4a6f401c8804a0ba6ea3b79f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d569aec4a6f401c8804a0ba6ea3b79f, State: Initialized, Role: FOLLOWER
I20260806 02:53:02.489418 6396 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d569aec4a6f401c8804a0ba6ea3b79f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d569aec4a6f401c8804a0ba6ea3b79f" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 35411 } }
I20260806 02:53:02.489688 6396 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d569aec4a6f401c8804a0ba6ea3b79f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:53:02.489841 6396 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d569aec4a6f401c8804a0ba6ea3b79f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:53:02.490033 6396 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d569aec4a6f401c8804a0ba6ea3b79f [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:53:02.492404 6396 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d569aec4a6f401c8804a0ba6ea3b79f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d569aec4a6f401c8804a0ba6ea3b79f" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 35411 } }
I20260806 02:53:02.492997 6396 leader_election.cc:302] T 00000000000000000000000000000000 P 4d569aec4a6f401c8804a0ba6ea3b79f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d569aec4a6f401c8804a0ba6ea3b79f; no voters:
I20260806 02:53:02.493736 6396 leader_election.cc:288] T 00000000000000000000000000000000 P 4d569aec4a6f401c8804a0ba6ea3b79f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:53:02.493837 6402 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d569aec4a6f401c8804a0ba6ea3b79f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:53:02.494599 6402 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d569aec4a6f401c8804a0ba6ea3b79f [term 1 LEADER]: Becoming Leader. State: Replica: 4d569aec4a6f401c8804a0ba6ea3b79f, State: Running, Role: LEADER
I20260806 02:53:02.495157 6402 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d569aec4a6f401c8804a0ba6ea3b79f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d569aec4a6f401c8804a0ba6ea3b79f" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 35411 } }
I20260806 02:53:02.495908 6396 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d569aec4a6f401c8804a0ba6ea3b79f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:53:02.497963 6404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d569aec4a6f401c8804a0ba6ea3b79f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d569aec4a6f401c8804a0ba6ea3b79f. Latest consensus state: current_term: 1 leader_uuid: "4d569aec4a6f401c8804a0ba6ea3b79f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d569aec4a6f401c8804a0ba6ea3b79f" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 35411 } } }
I20260806 02:53:02.498667 6404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d569aec4a6f401c8804a0ba6ea3b79f [sys.catalog]: This master's current role is: LEADER
I20260806 02:53:02.498297 6403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d569aec4a6f401c8804a0ba6ea3b79f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d569aec4a6f401c8804a0ba6ea3b79f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d569aec4a6f401c8804a0ba6ea3b79f" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 35411 } } }
I20260806 02:53:02.499557 6412 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:53:02.499598 6403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d569aec4a6f401c8804a0ba6ea3b79f [sys.catalog]: This master's current role is: LEADER
I20260806 02:53:02.505026 6412 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:53:02.515200 6412 catalog_manager.cc:1347] Generated new cluster ID: ef990d67756f49c99d60d28a9163ae6a
I20260806 02:53:02.515323 6412 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:53:02.530685 6412 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:53:02.532058 6412 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:53:02.543671 6412 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d569aec4a6f401c8804a0ba6ea3b79f: Generated new TSK 0
I20260806 02:53:02.544550 6412 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.211.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab.
Entry for principal kudu/127.25.211.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab.
I20260806 02:53:02.622745 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.129:0
--local_ip_for_outbound_sockets=127.25.211.129
--webserver_interface=127.25.211.129
--webserver_port=0
--tserver_master_addrs=127.25.211.190:35411
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab
--principal=kudu/127.25.211.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:38545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:53:02.983419 6434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:53:02.983716 6434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:53:02.983814 6434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:53:02.991645 6434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:53:02.991796 6434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:53:02.991914 6434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.129
I20260806 02:53:03.002373 6434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:38545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab
--principal=kudu/127.25.211.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.211.129
--webserver_port=0
--tserver_master_addrs=127.25.211.190:35411
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.129
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:53:03.003727 6434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:53:03.005079 6434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:53:03.019724 6442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:03.019748 6444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:03.020440 6450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:53:03.020560 6434 server_base.cc:1034] running on GCE node
I20260806 02:53:03.021217 6434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:53:03.022199 6434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:53:03.023425 6434 hybrid_clock.cc:648] HybridClock initialized: now 1785984783023379 us; error 38 us; skew 500 ppm
I20260806 02:53:03.023767 6434 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:53:03 dist-test-slave-zj8q krb5kdc[6263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984783, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:53:03.027166 6434 init.cc:377] Logged in from keytab as kudu/127.25.211.129@KRBTEST.COM (short username kudu)
I20260806 02:53:03.028998 6434 webserver.cc:460] Webserver started at http://127.25.211.129:33843/ using document root <none> and password file <none>
I20260806 02:53:03.029750 6434 fs_manager.cc:359] Metadata directory not provided
I20260806 02:53:03.029881 6434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:53:03.030279 6434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:53:03.033416 6434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-0/data/instance:
uuid: "10bbc1a322414a488e106388118d7c34"
format_stamp: "Formatted at 2026-08-06 02:53:03 on dist-test-slave-zj8q"
server_key: "a1c4ae1134f4a760c1ba82bea09a498b"
server_key_iv: "9f5043dc99bbea300adfa42ed91d012d"
server_key_version: "encryptionkey@0"
I20260806 02:53:03.034291 6434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-0/wal/instance:
uuid: "10bbc1a322414a488e106388118d7c34"
format_stamp: "Formatted at 2026-08-06 02:53:03 on dist-test-slave-zj8q"
server_key: "a1c4ae1134f4a760c1ba82bea09a498b"
server_key_iv: "9f5043dc99bbea300adfa42ed91d012d"
server_key_version: "encryptionkey@0"
I20260806 02:53:03.039882 6434 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260806 02:53:03.043840 6460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:03.045637 6434 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:53:03.045847 6434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-0/wal
uuid: "10bbc1a322414a488e106388118d7c34"
format_stamp: "Formatted at 2026-08-06 02:53:03 on dist-test-slave-zj8q"
server_key: "a1c4ae1134f4a760c1ba82bea09a498b"
server_key_iv: "9f5043dc99bbea300adfa42ed91d012d"
server_key_version: "encryptionkey@0"
I20260806 02:53:03.046072 6434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:53:03.058106 6434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:53:03.062858 6434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:53:03.063156 6434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:53:03.064209 6434 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:53:03.065691 6434 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:53:03.065780 6434 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:03.065907 6434 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:53:03.065963 6434 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:03.099262 6434 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.129:45679
I20260806 02:53:03.099313 6633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.129:45679 every 8 connection(s)
I20260806 02:53:03.100629 6434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
I20260806 02:53:03.103616 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 6434
I20260806 02:53:03.103907 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-0/wal/instance
I20260806 02:53:03.104565 26446 external_mini_cluster.cc:1428] Setting key 8bee843b1ede8d4aeb90a8948ab063a1
Aug 06 02:53:03 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984783, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.129@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:53:03.125170 6636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:03.103541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:37979 (local address 127.25.211.190:35411)
0806 02:53:03.104088 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:03.104095 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:03.105202 (+ 1107us) server_negotiation.cc:407] Connection header received
0806 02:53:03.107614 (+ 2412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:03.107621 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:03.107724 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:03.107852 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:03.109786 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:03.110788 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:03.112939 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:03.113483 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:03.113572 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:03.118381 (+ 4809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:03.118390 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:03.118394 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:53:03.118435 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:03.121590 (+ 3155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:03.122546 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:03.122552 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:03.122555 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:03.122717 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:03.123168 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:03.123173 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:03.123176 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:03.123613 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:03.123763 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:03.124542 (+ 779us) server_negotiation.cc:299] Negotiation successful
0806 02:53:03.124808 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":269,"threads_started":1}
I20260806 02:53:03.126052 6637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:03.104069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:35411 (local address 127.25.211.129:37979)
0806 02:53:03.104983 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:03.105036 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:03.107292 (+ 2256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:03.108190 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:03.108200 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:03.108827 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:03.109506 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:03.109528 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:03.110988 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:03.110993 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:03.112756 (+ 1763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:03.112779 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:03.113931 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:03.113937 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:03.114253 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:03.115064 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:03.115093 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:03.118022 (+ 2929us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:53:03.121807 (+ 3785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:03.121816 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:03.121833 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:03.122405 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:03.122826 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:03.122829 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:03.122831 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:03.123040 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:03.123746 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:53:03.123753 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:53:03.124113 (+ 360us) client_negotiation.cc:769] Sending connection context
0806 02:53:03.124371 (+ 258us) client_negotiation.cc:240] Negotiation successful
0806 02:53:03.124821 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":346,"threads_started":1}
I20260806 02:53:03.127861 6634 heartbeater.cc:347] Connected to a master server at 127.25.211.190:35411
I20260806 02:53:03.128245 6634 heartbeater.cc:464] Registering TS with master...
I20260806 02:53:03.129182 6634 heartbeater.cc:511] Master 127.25.211.190:35411 requested a full tablet report, sending...
I20260806 02:53:03.132052 6334 ts_manager.cc:194] Registered new tserver with Master: 10bbc1a322414a488e106388118d7c34 (127.25.211.129:45679)
I20260806 02:53:03.134292 6334 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.211.129@KRBTEST.COM'} at 127.25.211.129:37979
WARNING: no policy specified for kudu/127.25.211.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab.
Entry for principal kudu/127.25.211.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab.
I20260806 02:53:03.169924 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.130:0
--local_ip_for_outbound_sockets=127.25.211.130
--webserver_interface=127.25.211.130
--webserver_port=0
--tserver_master_addrs=127.25.211.190:35411
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab
--principal=kudu/127.25.211.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:38545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:53:03.521613 6640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:53:03.521921 6640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:53:03.522047 6640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:53:03.529611 6640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:53:03.529763 6640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:53:03.529918 6640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.130
I20260806 02:53:03.540213 6640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:38545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab
--principal=kudu/127.25.211.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.211.130
--webserver_port=0
--tserver_master_addrs=127.25.211.190:35411
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.130
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:53:03.541599 6640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:53:03.543095 6640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:53:03.557003 6649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:03.557107 6650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:03.557129 6652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:53:03.557767 6640 server_base.cc:1034] running on GCE node
I20260806 02:53:03.558619 6640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:53:03.559620 6640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:53:03.560904 6640 hybrid_clock.cc:648] HybridClock initialized: now 1785984783560829 us; error 61 us; skew 500 ppm
I20260806 02:53:03.561256 6640 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:53:03 dist-test-slave-zj8q krb5kdc[6263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984783, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:53:03.564523 6640 init.cc:377] Logged in from keytab as kudu/127.25.211.130@KRBTEST.COM (short username kudu)
I20260806 02:53:03.566092 6640 webserver.cc:460] Webserver started at http://127.25.211.130:38905/ using document root <none> and password file <none>
I20260806 02:53:03.566836 6640 fs_manager.cc:359] Metadata directory not provided
I20260806 02:53:03.566943 6640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:53:03.567224 6640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:53:03.570379 6640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-1/data/instance:
uuid: "bd61a992f1404e4195e40ef316498970"
format_stamp: "Formatted at 2026-08-06 02:53:03 on dist-test-slave-zj8q"
server_key: "b0daf84a7987511045031fc9b2e33efa"
server_key_iv: "99a34925312438f3983e2078b868fdbe"
server_key_version: "encryptionkey@0"
I20260806 02:53:03.571204 6640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-1/wal/instance:
uuid: "bd61a992f1404e4195e40ef316498970"
format_stamp: "Formatted at 2026-08-06 02:53:03 on dist-test-slave-zj8q"
server_key: "b0daf84a7987511045031fc9b2e33efa"
server_key_iv: "99a34925312438f3983e2078b868fdbe"
server_key_version: "encryptionkey@0"
I20260806 02:53:03.577368 6640 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260806 02:53:03.581281 6661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:03.583022 6640 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:53:03.583199 6640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-1/wal
uuid: "bd61a992f1404e4195e40ef316498970"
format_stamp: "Formatted at 2026-08-06 02:53:03 on dist-test-slave-zj8q"
server_key: "b0daf84a7987511045031fc9b2e33efa"
server_key_iv: "99a34925312438f3983e2078b868fdbe"
server_key_version: "encryptionkey@0"
I20260806 02:53:03.583391 6640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:53:03.605132 6640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:53:03.610047 6640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:53:03.610345 6640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:53:03.611410 6640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:53:03.612874 6640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:53:03.612953 6640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:03.613040 6640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:53:03.613085 6640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:03.647924 6640 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.130:36333
I20260806 02:53:03.647987 6826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.130:36333 every 8 connection(s)
I20260806 02:53:03.649343 6640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
I20260806 02:53:03.659827 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 6640
I20260806 02:53:03.659983 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-1/wal/instance
I20260806 02:53:03.660429 26446 external_mini_cluster.cc:1428] Setting key 9af0d26053ad7b3a6f2935e398c914d0
Aug 06 02:53:03 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984783, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.130@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:53:03.672250 6830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:03.652754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:53647 (local address 127.25.211.190:35411)
0806 02:53:03.653295 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:03.653300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:03.654251 (+ 951us) server_negotiation.cc:407] Connection header received
0806 02:53:03.656288 (+ 2037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:03.656293 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:03.656376 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:03.656484 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:03.658330 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:03.659107 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:03.661108 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:03.661604 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:03.661741 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:03.665905 (+ 4164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:03.665911 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:03.665915 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:53:03.665955 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:03.668652 (+ 2697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:03.669595 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:03.669600 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:03.669602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:03.669763 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:03.670310 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:03.670314 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:03.670317 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:03.670627 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:03.670780 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:03.671591 (+ 811us) server_negotiation.cc:299] Negotiation successful
0806 02:53:03.671866 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":292,"threads_started":1}
I20260806 02:53:03.673472 6832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:03.653217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:35411 (local address 127.25.211.130:53647)
0806 02:53:03.654015 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:03.654070 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:03.655938 (+ 1868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:03.656738 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:03.656747 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:03.657386 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:03.658045 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:03.658066 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:03.659314 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:03.659318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:03.660911 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:03.660927 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:03.661875 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:03.661884 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:03.662247 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:03.663085 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:03.663119 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:03.665586 (+ 2467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:53:03.668875 (+ 3289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:03.668886 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:03.668904 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:03.669437 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:03.669899 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:03.669904 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:03.669908 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:03.670132 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:03.670776 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:53:03.670786 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:53:03.671191 (+ 405us) client_negotiation.cc:769] Sending connection context
0806 02:53:03.671452 (+ 261us) client_negotiation.cc:240] Negotiation successful
0806 02:53:03.672052 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":301,"threads_started":1}
I20260806 02:53:03.675274 6827 heartbeater.cc:347] Connected to a master server at 127.25.211.190:35411
I20260806 02:53:03.675631 6827 heartbeater.cc:464] Registering TS with master...
I20260806 02:53:03.676561 6827 heartbeater.cc:511] Master 127.25.211.190:35411 requested a full tablet report, sending...
I20260806 02:53:03.678532 6334 ts_manager.cc:194] Registered new tserver with Master: bd61a992f1404e4195e40ef316498970 (127.25.211.130:36333)
I20260806 02:53:03.679641 6334 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.211.130@KRBTEST.COM'} at 127.25.211.130:53647
WARNING: no policy specified for kudu/127.25.211.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab.
Entry for principal kudu/127.25.211.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab.
I20260806 02:53:03.730450 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.131:0
--local_ip_for_outbound_sockets=127.25.211.131
--webserver_interface=127.25.211.131
--webserver_port=0
--tserver_master_addrs=127.25.211.190:35411
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab
--principal=kudu/127.25.211.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:38545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:53:04.099723 6835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:53:04.100003 6835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:53:04.100273 6835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:53:04.108034 6835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:53:04.108165 6835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:53:04.108332 6835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.131
I20260806 02:53:04.119122 6835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:38545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab
--principal=kudu/127.25.211.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.211.131
--webserver_port=0
--tserver_master_addrs=127.25.211.190:35411
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.131
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:53:04.120574 6835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:53:04.121984 6835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:53:04.136713 6845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:04.136629 6849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:04.136663 6846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:53:04.137435 6835 server_base.cc:1034] running on GCE node
I20260806 02:53:04.138046 6634 heartbeater.cc:503] Master 127.25.211.190:35411 was elected leader, sending a full tablet report...
I20260806 02:53:04.138235 6835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:53:04.139508 6835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:53:04.140759 6835 hybrid_clock.cc:648] HybridClock initialized: now 1785984784140702 us; error 54 us; skew 500 ppm
I20260806 02:53:04.141105 6835 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:53:04 dist-test-slave-zj8q krb5kdc[6263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984784, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:53:04.149060 6835 init.cc:377] Logged in from keytab as kudu/127.25.211.131@KRBTEST.COM (short username kudu)
I20260806 02:53:04.150903 6835 webserver.cc:460] Webserver started at http://127.25.211.131:34515/ using document root <none> and password file <none>
I20260806 02:53:04.151657 6835 fs_manager.cc:359] Metadata directory not provided
I20260806 02:53:04.151783 6835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:53:04.152117 6835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:53:04.155359 6835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-2/data/instance:
uuid: "806f97f58aa7476e87bdd07c88cfa8a6"
format_stamp: "Formatted at 2026-08-06 02:53:04 on dist-test-slave-zj8q"
server_key: "997f91ef4d218e19a992f8d285eff419"
server_key_iv: "dab55237da05d706d0c24f3fd420eed7"
server_key_version: "encryptionkey@0"
I20260806 02:53:04.156208 6835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-2/wal/instance:
uuid: "806f97f58aa7476e87bdd07c88cfa8a6"
format_stamp: "Formatted at 2026-08-06 02:53:04 on dist-test-slave-zj8q"
server_key: "997f91ef4d218e19a992f8d285eff419"
server_key_iv: "dab55237da05d706d0c24f3fd420eed7"
server_key_version: "encryptionkey@0"
I20260806 02:53:04.162312 6835 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.002s
I20260806 02:53:04.166247 6862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:04.167713 6835 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:53:04.167925 6835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-2/wal
uuid: "806f97f58aa7476e87bdd07c88cfa8a6"
format_stamp: "Formatted at 2026-08-06 02:53:04 on dist-test-slave-zj8q"
server_key: "997f91ef4d218e19a992f8d285eff419"
server_key_iv: "dab55237da05d706d0c24f3fd420eed7"
server_key_version: "encryptionkey@0"
I20260806 02:53:04.168138 6835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:53:04.196754 6835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:53:04.201478 6835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:53:04.201810 6835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:53:04.202907 6835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:53:04.204396 6835 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:53:04.204488 6835 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:04.204617 6835 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:53:04.204682 6835 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:04.237990 6835 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.131:39455
I20260806 02:53:04.238037 7033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.131:39455 every 8 connection(s)
I20260806 02:53:04.239387 6835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
I20260806 02:53:04.241590 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 6835
I20260806 02:53:04.241752 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-2/wal/instance
I20260806 02:53:04.242339 26446 external_mini_cluster.cc:1428] Setting key b355bbc5670ba43383b8d2f8afc5de33
Aug 06 02:53:04 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984784, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.131@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:53:04.263432 7037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:04.242148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:33767 (local address 127.25.211.190:35411)
0806 02:53:04.242683 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:04.242690 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:04.243966 (+ 1276us) server_negotiation.cc:407] Connection header received
0806 02:53:04.246341 (+ 2375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:04.246347 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:04.246446 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:04.246539 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:04.248258 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:04.249047 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:04.250895 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:04.251302 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:04.251417 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:04.255720 (+ 4303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:04.255726 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:04.255729 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:53:04.255769 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:04.259146 (+ 3377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:04.260298 (+ 1152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:04.260305 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:04.260309 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:04.260458 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:04.261064 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:04.261070 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:04.261074 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:04.261470 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:04.261657 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:04.262748 (+ 1091us) server_negotiation.cc:299] Negotiation successful
0806 02:53:04.263026 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":225,"threads_started":1}
I20260806 02:53:04.264556 7038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:04.242831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:35411 (local address 127.25.211.131:33767)
0806 02:53:04.243774 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:04.243830 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:04.245976 (+ 2146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:04.246841 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:04.246850 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:04.247428 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:04.248059 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:04.248079 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:04.249238 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:04.249242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:04.250742 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:04.250756 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:04.251557 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:04.251565 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:04.251823 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:04.252536 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:04.252562 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:04.255407 (+ 2845us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:53:04.259341 (+ 3934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:04.259350 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:04.259367 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:04.260115 (+ 748us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:04.260598 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:04.260601 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:04.260604 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:04.260904 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:04.261665 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:53:04.261692 (+ 27us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:53:04.262290 (+ 598us) client_negotiation.cc:769] Sending connection context
0806 02:53:04.262564 (+ 274us) client_negotiation.cc:240] Negotiation successful
0806 02:53:04.263156 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"thread_start_us":404,"threads_started":1}
I20260806 02:53:04.266644 7035 heartbeater.cc:347] Connected to a master server at 127.25.211.190:35411
I20260806 02:53:04.267056 7035 heartbeater.cc:464] Registering TS with master...
I20260806 02:53:04.268292 7035 heartbeater.cc:511] Master 127.25.211.190:35411 requested a full tablet report, sending...
I20260806 02:53:04.272185 6334 ts_manager.cc:194] Registered new tserver with Master: 806f97f58aa7476e87bdd07c88cfa8a6 (127.25.211.131:39455)
I20260806 02:53:04.273241 6334 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.211.131@KRBTEST.COM'} at 127.25.211.131:33767
I20260806 02:53:04.274962 26446 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.25.211.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.211.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.211.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/krb5kdc/oryx_127.25.211.129.keytab.
Entry for principal oryx/127.25.211.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/krb5kdc/oryx_127.25.211.129.keytab.
WARNING: no policy specified for oryx/127.25.211.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.211.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.211.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/krb5kdc/oryx_127.25.211.130.keytab.
Entry for principal oryx/127.25.211.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/krb5kdc/oryx_127.25.211.130.keytab.
WARNING: no policy specified for oryx/127.25.211.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.25.211.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.25.211.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/krb5kdc/oryx_127.25.211.131.keytab.
Entry for principal oryx/127.25.211.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/krb5kdc/oryx_127.25.211.131.keytab.
I20260806 02:53:04.474251 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 6434
Aug 06 02:53:04 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.129@KRBTEST.COM
I20260806 02:53:04.492895 7051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:04.475711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41736 (local address 127.25.211.129:45679)
0806 02:53:04.476423 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:04.476430 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:04.476459 (+ 29us) server_negotiation.cc:407] Connection header received
0806 02:53:04.476524 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:04.476531 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:04.476755 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:04.476901 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:04.478492 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:04.479618 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:04.481050 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:04.481665 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:04.481809 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:04.486463 (+ 4654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:04.486481 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:04.486500 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:53:04.486552 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:04.489824 (+ 3272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:04.490755 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:04.490767 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:04.490770 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:04.490965 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:04.491390 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:04.491394 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:04.491396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:04.491788 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:04.492060 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:04.492285 (+ 225us) server_negotiation.cc:299] Negotiation successful
0806 02:53:04.492511 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":387,"threads_started":1}
I20260806 02:53:04.494290 6627 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:53:04.580479 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 6434
I20260806 02:53:04.600246 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 6640
Aug 06 02:53:04 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.130@KRBTEST.COM
I20260806 02:53:04.617607 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:04.601131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55740 (local address 127.25.211.130:36333)
0806 02:53:04.601757 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:04.601766 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:04.601794 (+ 28us) server_negotiation.cc:407] Connection header received
0806 02:53:04.601859 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:04.601865 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:04.602074 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:04.602267 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:04.603534 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:04.604617 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:04.606053 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:04.606951 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:04.607069 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:04.611697 (+ 4628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:04.611714 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:04.611731 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:53:04.611783 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:04.614794 (+ 3011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:04.615522 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:04.615529 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:04.615531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:04.615668 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:04.615989 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:04.615992 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:04.615994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:04.616360 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:04.616589 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:04.616993 (+ 404us) server_negotiation.cc:299] Negotiation successful
0806 02:53:04.617208 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":294,"threads_started":1}
I20260806 02:53:04.619094 6821 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:53:04.701658 6827 heartbeater.cc:503] Master 127.25.211.190:35411 was elected leader, sending a full tablet report...
I20260806 02:53:04.702834 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 6640
I20260806 02:53:04.729902 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 6835
Aug 06 02:53:04 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.131@KRBTEST.COM
I20260806 02:53:04.749010 7055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:04.731207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55060 (local address 127.25.211.131:39455)
0806 02:53:04.731826 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:04.731833 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:04.731868 (+ 35us) server_negotiation.cc:407] Connection header received
0806 02:53:04.731938 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:04.731945 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:04.732178 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:04.732356 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:04.733897 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:04.735028 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:04.736448 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:04.737073 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:04.737202 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:04.742261 (+ 5059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:04.742283 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:04.742303 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:53:04.742350 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:04.745772 (+ 3422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:04.746685 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:04.746693 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:04.746696 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:04.746836 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:04.747278 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:04.747282 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:04.747284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:04.747682 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:04.747992 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:04.748383 (+ 391us) server_negotiation.cc:299] Negotiation successful
0806 02:53:04.748662 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"spinlock_wait_cycles":25344,"thread_start_us":307,"threads_started":1}
I20260806 02:53:04.750461 7029 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:53:04.838497 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 6835
I20260806 02:53:04.857973 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 6286
I20260806 02:53:04.859189 6389 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:53:04.966322 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 6286
I20260806 02:53:04.992658 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:35411
--webserver_interface=127.25.211.190
--webserver_port=42571
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:38545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.211.190:35411
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:53:05.353426 7062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:53:05.353806 7062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:53:05.353948 7062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:53:05.361869 7062 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:53:05.362017 7062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:53:05.362066 7062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:53:05.362147 7062 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:53:05.362233 7062 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:53:05.372771 7062 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:38545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.211.190:35411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:35411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.211.190
--webserver_port=42571
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:53:05.374312 7062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:53:05.375735 7062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:53:05.388516 7074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:05.388484 7081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:05.388561 7078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:53:05.389487 7062 server_base.cc:1034] running on GCE node
I20260806 02:53:05.390453 7062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:53:05.392141 7062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:53:05.393392 7062 hybrid_clock.cc:648] HybridClock initialized: now 1785984785393326 us; error 64 us; skew 500 ppm
I20260806 02:53:05.393803 7062 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:53:05 dist-test-slave-zj8q krb5kdc[6263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984785, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:53:05.399499 7062 init.cc:377] Logged in from keytab as kudu/127.25.211.190@KRBTEST.COM (short username kudu)
I20260806 02:53:05.401607 7062 webserver.cc:460] Webserver started at http://127.25.211.190:42571/ using document root <none> and password file <none>
I20260806 02:53:05.402683 7062 fs_manager.cc:359] Metadata directory not provided
I20260806 02:53:05.402906 7062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:53:05.409106 7062 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:53:05.412295 7089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:05.413710 7062 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:53:05.413883 7062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/master-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/master-0/wal
uuid: "4d569aec4a6f401c8804a0ba6ea3b79f"
format_stamp: "Formatted at 2026-08-06 02:53:02 on dist-test-slave-zj8q"
server_key: "0413d612db97780341f0aaffef5e8d25"
server_key_iv: "17602882ec43c7f5abaa658f3585978f"
server_key_version: "encryptionkey@0"
I20260806 02:53:05.414937 7062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:53:05.449980 7062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:53:05.454730 7062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:53:05.455013 7062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:53:05.474417 7062 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.190:35411
I20260806 02:53:05.474437 7168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.190:35411 every 8 connection(s)
I20260806 02:53:05.476011 7062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
I20260806 02:53:05.483461 7170 sys_catalog.cc:263] Verifying existing consensus state
I20260806 02:53:05.483619 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 7062
I20260806 02:53:05.484615 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.129:45679
--local_ip_for_outbound_sockets=127.25.211.129
--tserver_master_addrs=127.25.211.190:35411
--webserver_port=33843
--webserver_interface=127.25.211.129
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab
--principal=kudu/127.25.211.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:38545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.25.211.129
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/krb5kdc/oryx_127.25.211.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:53:05.488276 7170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d569aec4a6f401c8804a0ba6ea3b79f: Bootstrap starting.
I20260806 02:53:05.510756 7170 log.cc:824] T 00000000000000000000000000000000 P 4d569aec4a6f401c8804a0ba6ea3b79f: Log is configured to *not* fsync() on all Append() calls
I20260806 02:53:05.520942 7170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d569aec4a6f401c8804a0ba6ea3b79f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:53:05.521556 7170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d569aec4a6f401c8804a0ba6ea3b79f: Bootstrap complete.
I20260806 02:53:05.531034 7170 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d569aec4a6f401c8804a0ba6ea3b79f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d569aec4a6f401c8804a0ba6ea3b79f" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 35411 } }
I20260806 02:53:05.532078 7170 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d569aec4a6f401c8804a0ba6ea3b79f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d569aec4a6f401c8804a0ba6ea3b79f, State: Initialized, Role: FOLLOWER
I20260806 02:53:05.532739 7170 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d569aec4a6f401c8804a0ba6ea3b79f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d569aec4a6f401c8804a0ba6ea3b79f" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 35411 } }
I20260806 02:53:05.532985 7170 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d569aec4a6f401c8804a0ba6ea3b79f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:53:05.533106 7170 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d569aec4a6f401c8804a0ba6ea3b79f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:53:05.533288 7170 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d569aec4a6f401c8804a0ba6ea3b79f [term 1 FOLLOWER]: Advancing to term 2
I20260806 02:53:05.537364 7170 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d569aec4a6f401c8804a0ba6ea3b79f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d569aec4a6f401c8804a0ba6ea3b79f" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 35411 } }
I20260806 02:53:05.537915 7170 leader_election.cc:302] T 00000000000000000000000000000000 P 4d569aec4a6f401c8804a0ba6ea3b79f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d569aec4a6f401c8804a0ba6ea3b79f; no voters:
I20260806 02:53:05.538573 7170 leader_election.cc:288] T 00000000000000000000000000000000 P 4d569aec4a6f401c8804a0ba6ea3b79f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 02:53:05.538718 7174 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d569aec4a6f401c8804a0ba6ea3b79f [term 2 FOLLOWER]: Leader election won for term 2
I20260806 02:53:05.539870 7174 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d569aec4a6f401c8804a0ba6ea3b79f [term 2 LEADER]: Becoming Leader. State: Replica: 4d569aec4a6f401c8804a0ba6ea3b79f, State: Running, Role: LEADER
I20260806 02:53:05.540536 7174 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d569aec4a6f401c8804a0ba6ea3b79f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d569aec4a6f401c8804a0ba6ea3b79f" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 35411 } }
I20260806 02:53:05.540874 7170 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d569aec4a6f401c8804a0ba6ea3b79f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:53:05.543361 7177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d569aec4a6f401c8804a0ba6ea3b79f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d569aec4a6f401c8804a0ba6ea3b79f. Latest consensus state: current_term: 2 leader_uuid: "4d569aec4a6f401c8804a0ba6ea3b79f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d569aec4a6f401c8804a0ba6ea3b79f" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 35411 } } }
I20260806 02:53:05.543856 7177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d569aec4a6f401c8804a0ba6ea3b79f [sys.catalog]: This master's current role is: LEADER
I20260806 02:53:05.544500 7175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d569aec4a6f401c8804a0ba6ea3b79f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4d569aec4a6f401c8804a0ba6ea3b79f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d569aec4a6f401c8804a0ba6ea3b79f" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 35411 } } }
I20260806 02:53:05.544863 7175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d569aec4a6f401c8804a0ba6ea3b79f [sys.catalog]: This master's current role is: LEADER
I20260806 02:53:05.549446 7183 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:53:05.558820 7183 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:53:05.560545 7183 catalog_manager.cc:1259] Loaded cluster ID: ef990d67756f49c99d60d28a9163ae6a
I20260806 02:53:05.560761 7183 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:53:05.565255 7183 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:53:05.565763 7196 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 02:53:05.567160 7183 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4d569aec4a6f401c8804a0ba6ea3b79f: Loaded TSK: 0
I20260806 02:53:05.568619 7183 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 02:53:05.863716 7172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:53:05.864055 7172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:53:05.864197 7172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:53:05.872460 7172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:53:05.872591 7172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:53:05.872753 7172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.129
I20260806 02:53:05.883455 7172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:38545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/krb5kdc/oryx_127.25.211.129.keytab
--principal=oryx/127.25.211.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.129:45679
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.211.129
--webserver_port=33843
--tserver_master_addrs=127.25.211.190:35411
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.129
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:53:05.885027 7172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:53:05.886914 7172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:53:05.902810 7204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:05.903167 7201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:05.903031 7202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:53:05.904022 7172 server_base.cc:1034] running on GCE node
I20260806 02:53:05.904735 7172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:53:05.906045 7172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:53:05.907438 7172 hybrid_clock.cc:648] HybridClock initialized: now 1785984785907372 us; error 58 us; skew 500 ppm
I20260806 02:53:05.907812 7172 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:53:05 dist-test-slave-zj8q krb5kdc[6263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984785, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.211.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:53:05.911384 7172 init.cc:377] Logged in from keytab as oryx/127.25.211.129@KRBTEST.COM (short username oryx)
I20260806 02:53:05.913130 7172 webserver.cc:460] Webserver started at http://127.25.211.129:33843/ using document root <none> and password file <none>
I20260806 02:53:05.913861 7172 fs_manager.cc:359] Metadata directory not provided
I20260806 02:53:05.914005 7172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:53:05.920054 7172 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260806 02:53:05.923152 7212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:05.924662 7172 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:53:05.924845 7172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-0/wal
uuid: "10bbc1a322414a488e106388118d7c34"
format_stamp: "Formatted at 2026-08-06 02:53:03 on dist-test-slave-zj8q"
server_key: "a1c4ae1134f4a760c1ba82bea09a498b"
server_key_iv: "9f5043dc99bbea300adfa42ed91d012d"
server_key_version: "encryptionkey@0"
I20260806 02:53:05.925971 7172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:53:05.943683 7172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:53:05.948177 7172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:53:05.948450 7172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:53:05.949496 7172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:53:05.950971 7172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:53:05.951071 7172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:05.951180 7172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:53:05.951252 7172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:05.982916 7172 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.129:45679
I20260806 02:53:05.983021 7367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.129:45679 every 8 connection(s)
I20260806 02:53:05.984417 7172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
I20260806 02:53:05.986878 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 7172
I20260806 02:53:05.987608 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.130:36333
--local_ip_for_outbound_sockets=127.25.211.130
--tserver_master_addrs=127.25.211.190:35411
--webserver_port=38905
--webserver_interface=127.25.211.130
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab
--principal=kudu/127.25.211.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:38545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.25.211.130
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/krb5kdc/oryx_127.25.211.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:53:06 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.129@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:53:06 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.129@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:53:06.008070 7374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:05.987946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:35411 (local address 127.25.211.129:34765)
0806 02:53:05.990816 (+ 2870us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:05.990877 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:05.992709 (+ 1832us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:05.994579 (+ 1870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:05.994587 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:05.995185 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:05.995751 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:05.995770 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:05.997944 (+ 2174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:05.997948 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:05.999919 (+ 1971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:05.999936 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:06.001153 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:06.001160 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:06.001453 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:06.002270 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:06.002299 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:06.006568 (+ 4269us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.211.190:35411: Server oryx/127.25.211.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":2536,"thread_start_us":2308,"threads_started":1}
W20260806 02:53:06.008100 7370 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:53:05.987933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:34765 (local address 127.25.211.190:35411)
0806 02:53:05.988761 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:05.988779 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:05.991056 (+ 2277us) server_negotiation.cc:407] Connection header received
0806 02:53:05.993431 (+ 2375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:05.993452 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:05.993966 (+ 514us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:05.994419 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:05.995978 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:05.997739 (+ 1761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:06.000114 (+ 2375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:06.000874 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:06.001034 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:06.007038 (+ 6004us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.211.129:34765: BlockingRecv error: recv got EOF from 127.25.211.129:34765 (error 108)
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":295,"threads_started":1}
W20260806 02:53:06.009153 7368 heartbeater.cc:644] Failed to heartbeat to 127.25.211.190:35411 (0 consecutive failures): Not authorized: Failed to ping master at 127.25.211.190:35411: Client connection negotiation failed: client connection to 127.25.211.190:35411: Server oryx/127.25.211.190@KRBTEST.COM not found in Kerberos database
Aug 06 02:53:06 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.129@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:53:06 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.129@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:53:06.019924 7374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:06.009790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:35411 (local address 127.25.211.129:42759)
0806 02:53:06.009986 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:06.010013 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:06.010272 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:06.010644 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:06.010649 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:06.011041 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:06.011220 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:06.011229 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:06.012491 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:06.012497 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:06.013531 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:06.013546 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:06.014517 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:06.014523 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:06.014827 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:06.015478 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:06.015502 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:06.019612 (+ 4110us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.211.190:35411: Server oryx/127.25.211.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260806 02:53:06.020521 7370 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:53:06.009928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:42759 (local address 127.25.211.190:35411)
0806 02:53:06.010251 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:06.010257 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:06.010283 (+ 26us) server_negotiation.cc:407] Connection header received
0806 02:53:06.010428 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:06.010434 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:06.010498 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:06.010588 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:06.011389 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:06.012311 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:06.013687 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:06.014334 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:06.014467 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:06.020100 (+ 5633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.211.129:42759: BlockingRecv error: recv got EOF from 127.25.211.129:42759 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Aug 06 02:53:06 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.129@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:53:06 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.129@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:53:06.030537 7374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:06.020808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:35411 (local address 127.25.211.129:44639)
0806 02:53:06.021004 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:06.021020 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:06.021124 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:06.021779 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:06.021784 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:06.022227 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:06.022444 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:06.022457 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:06.023547 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:06.023553 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:06.025089 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:06.025105 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:06.025966 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:06.025973 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:06.026232 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:06.026816 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:06.026839 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:06.030229 (+ 3390us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.211.190:35411: Server oryx/127.25.211.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":86}
W20260806 02:53:06.030754 7370 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:53:06.021319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:44639 (local address 127.25.211.190:35411)
0806 02:53:06.021479 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:06.021483 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:06.021498 (+ 15us) server_negotiation.cc:407] Connection header received
0806 02:53:06.021547 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:06.021551 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:06.021619 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:06.021698 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:06.022626 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:06.023396 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:06.025272 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:06.025796 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:06.025896 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:06.030499 (+ 4603us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.211.129:44639: BlockingRecv error: recv got EOF from 127.25.211.129:44639 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260806 02:53:06.031049 7368 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 02:53:06.346084 7375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:53:06.346387 7375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:53:06.346521 7375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:53:06.353826 7375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:53:06.353940 7375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:53:06.354084 7375 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.130
I20260806 02:53:06.364224 7375 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:38545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/krb5kdc/oryx_127.25.211.130.keytab
--principal=oryx/127.25.211.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.130:36333
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.211.130
--webserver_port=38905
--tserver_master_addrs=127.25.211.190:35411
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.130
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:53:06.365635 7375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:53:06.367075 7375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:53:06.380414 7388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:06.380582 7383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:06.380532 7384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:53:06.381569 7375 server_base.cc:1034] running on GCE node
I20260806 02:53:06.382279 7375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:53:06.383278 7375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:53:06.384482 7375 hybrid_clock.cc:648] HybridClock initialized: now 1785984786384406 us; error 72 us; skew 500 ppm
I20260806 02:53:06.384800 7375 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:53:06 dist-test-slave-zj8q krb5kdc[6263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984786, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.211.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:53:06.388168 7375 init.cc:377] Logged in from keytab as oryx/127.25.211.130@KRBTEST.COM (short username oryx)
I20260806 02:53:06.389777 7375 webserver.cc:460] Webserver started at http://127.25.211.130:38905/ using document root <none> and password file <none>
I20260806 02:53:06.390554 7375 fs_manager.cc:359] Metadata directory not provided
I20260806 02:53:06.390691 7375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:53:06.396776 7375 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:53:06.400209 7396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:06.401875 7375 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:53:06.402223 7375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-1/wal
uuid: "bd61a992f1404e4195e40ef316498970"
format_stamp: "Formatted at 2026-08-06 02:53:03 on dist-test-slave-zj8q"
server_key: "b0daf84a7987511045031fc9b2e33efa"
server_key_iv: "99a34925312438f3983e2078b868fdbe"
server_key_version: "encryptionkey@0"
I20260806 02:53:06.403512 7375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:53:06.419921 7375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:53:06.424360 7375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:53:06.424633 7375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:53:06.425653 7375 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:53:06.427100 7375 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:53:06.427189 7375 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:06.427296 7375 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:53:06.427366 7375 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:06.458750 7375 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.130:36333
I20260806 02:53:06.458844 7570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.130:36333 every 8 connection(s)
I20260806 02:53:06.460331 7375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
I20260806 02:53:06.460788 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 7375
I20260806 02:53:06.461562 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.131:39455
--local_ip_for_outbound_sockets=127.25.211.131
--tserver_master_addrs=127.25.211.190:35411
--webserver_port=34515
--webserver_interface=127.25.211.131
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab
--principal=kudu/127.25.211.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:38545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.25.211.131
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/krb5kdc/oryx_127.25.211.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:53:06 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.130@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:53:06 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.130@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
W20260806 02:53:06.478740 7370 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:53:06.463369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:48105 (local address 127.25.211.190:35411)
0806 02:53:06.463593 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:06.463600 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:06.464878 (+ 1278us) server_negotiation.cc:407] Connection header received
0806 02:53:06.466837 (+ 1959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:06.466843 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:06.466929 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:06.467046 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:06.468687 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:06.469672 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:06.471512 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:06.472084 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:06.472234 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:06.478482 (+ 6248us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.211.130:48105: BlockingRecv error: recv got EOF from 127.25.211.130:48105 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 02:53:06.479583 7575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:06.463846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:35411 (local address 127.25.211.130:48105)
0806 02:53:06.464667 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:06.464735 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:06.466547 (+ 1812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:06.467242 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:06.467250 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:06.467859 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:06.468467 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:06.468486 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:06.469871 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:06.469875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:06.471339 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:06.471355 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:06.472408 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:06.472416 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:06.472739 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:06.473548 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:06.473579 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:06.478236 (+ 4657us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.211.190:35411: Server oryx/127.25.211.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":341,"threads_started":1}
W20260806 02:53:06.480365 7571 heartbeater.cc:644] Failed to heartbeat to 127.25.211.190:35411 (0 consecutive failures): Not authorized: Failed to ping master at 127.25.211.190:35411: Client connection negotiation failed: client connection to 127.25.211.190:35411: Server oryx/127.25.211.190@KRBTEST.COM not found in Kerberos database
Aug 06 02:53:06 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.130@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:53:06 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.130@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:53:06.490603 7575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:06.481005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:35411 (local address 127.25.211.130:58611)
0806 02:53:06.481278 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:06.481297 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:06.481414 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:06.481768 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:06.481773 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:06.482240 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:06.482474 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:06.482489 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:06.483829 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:06.483834 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:06.485336 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:06.485353 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:06.486408 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:06.486414 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:06.486588 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:06.487244 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:06.487264 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:06.490375 (+ 3111us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.211.190:35411: Server oryx/127.25.211.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130}
W20260806 02:53:06.491318 7370 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:53:06.481114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:58611 (local address 127.25.211.190:35411)
0806 02:53:06.481291 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:06.481297 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:06.481367 (+ 70us) server_negotiation.cc:407] Connection header received
0806 02:53:06.481533 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:06.481539 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:06.481617 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:06.481722 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:06.482643 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:06.483655 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:06.485500 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:06.486191 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:06.486340 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:06.490727 (+ 4387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.211.130:58611: BlockingRecv error: recv got EOF from 127.25.211.130:58611 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 06 02:53:06 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.130@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:53:06 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.130@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:53:06.499852 7575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:06.491312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:35411 (local address 127.25.211.130:51941)
0806 02:53:06.491555 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:06.491574 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:06.491716 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:06.492113 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:06.492119 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:06.492487 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:06.492642 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:06.492651 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:06.494079 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:06.494083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:06.495051 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:06.495064 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:06.496039 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:06.496045 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:06.496249 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:06.496841 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:06.496858 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:06.499615 (+ 2757us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.211.190:35411: Server oryx/127.25.211.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142}
W20260806 02:53:06.500136 7370 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:53:06.491509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:51941 (local address 127.25.211.190:35411)
0806 02:53:06.491801 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:06.491806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:06.491826 (+ 20us) server_negotiation.cc:407] Connection header received
0806 02:53:06.491883 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:06.491888 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:06.491967 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:06.492080 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:06.492786 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:06.493907 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:06.495180 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:06.495880 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:06.496001 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:06.499883 (+ 3882us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.211.130:51941: BlockingRecv error: recv got EOF from 127.25.211.130:51941 (error 108)
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":806,"threads_started":1}
W20260806 02:53:06.500253 7571 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 02:53:06.824187 7577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:53:06.824497 7577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:53:06.824615 7577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:53:06.832613 7577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:53:06.832762 7577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:53:06.832912 7577 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.131
I20260806 02:53:06.843384 7577 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:38545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/krb5kdc/oryx_127.25.211.131.keytab
--principal=oryx/127.25.211.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.131:39455
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.211.131
--webserver_port=34515
--tserver_master_addrs=127.25.211.190:35411
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.131
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:53:06.844798 7577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:53:06.846261 7577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:53:06.861747 7589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:06.862022 7590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:06.862604 7594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:53:06.863312 7577 server_base.cc:1034] running on GCE node
I20260806 02:53:06.864048 7577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:53:06.865036 7577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:53:06.866262 7577 hybrid_clock.cc:648] HybridClock initialized: now 1785984786866207 us; error 39 us; skew 500 ppm
I20260806 02:53:06.866598 7577 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:53:06 dist-test-slave-zj8q krb5kdc[6263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984786, etypes {rep=17 tkt=17 ses=17}, oryx/127.25.211.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:53:06.870292 7577 init.cc:377] Logged in from keytab as oryx/127.25.211.131@KRBTEST.COM (short username oryx)
I20260806 02:53:06.871927 7577 webserver.cc:460] Webserver started at http://127.25.211.131:34515/ using document root <none> and password file <none>
I20260806 02:53:06.872687 7577 fs_manager.cc:359] Metadata directory not provided
I20260806 02:53:06.872848 7577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:53:06.878569 7577 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260806 02:53:06.881424 7603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:06.883751 7577 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 02:53:06.884107 7577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-2/wal
uuid: "806f97f58aa7476e87bdd07c88cfa8a6"
format_stamp: "Formatted at 2026-08-06 02:53:04 on dist-test-slave-zj8q"
server_key: "997f91ef4d218e19a992f8d285eff419"
server_key_iv: "dab55237da05d706d0c24f3fd420eed7"
server_key_version: "encryptionkey@0"
I20260806 02:53:06.885296 7577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:53:06.900331 7577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:53:06.904626 7577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:53:06.904911 7577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:53:06.905951 7577 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:53:06.907486 7577 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:53:06.907591 7577 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:06.907711 7577 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:53:06.907784 7577 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:06.942387 7577 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.131:39455
I20260806 02:53:06.942472 7774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.131:39455 every 8 connection(s)
I20260806 02:53:06.943915 7577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
I20260806 02:53:06.947541 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 7577
Aug 06 02:53:06 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.131@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:53:06 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.131@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
W20260806 02:53:06.962023 7370 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:53:06.946655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:43645 (local address 127.25.211.190:35411)
0806 02:53:06.946838 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:06.946842 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:06.948167 (+ 1325us) server_negotiation.cc:407] Connection header received
0806 02:53:06.950460 (+ 2293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:06.950466 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:06.950597 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:06.950701 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:06.952602 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:06.953738 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:06.955812 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:06.956283 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:06.956382 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:06.961756 (+ 5374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.211.131:43645: BlockingRecv error: recv got EOF from 127.25.211.131:43645 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 02:53:06.962880 7779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:06.947170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:35411 (local address 127.25.211.131:43645)
0806 02:53:06.947979 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:06.948027 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:06.950174 (+ 2147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:06.950920 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:06.950929 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:06.951564 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:06.952368 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:06.952393 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:06.953906 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:06.953910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:06.955662 (+ 1752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:06.955676 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:06.956527 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:06.956535 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:06.956891 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:06.957576 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:06.957608 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:06.961505 (+ 3897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.211.190:35411: Server oryx/127.25.211.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":318,"threads_started":1}
W20260806 02:53:06.963577 7775 heartbeater.cc:644] Failed to heartbeat to 127.25.211.190:35411 (0 consecutive failures): Not authorized: Failed to ping master at 127.25.211.190:35411: Client connection negotiation failed: client connection to 127.25.211.190:35411: Server oryx/127.25.211.190@KRBTEST.COM not found in Kerberos database
I20260806 02:53:06.965178 7578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:06.948583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37628 (local address 127.25.211.190:35411)
0806 02:53:06.948791 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:06.948797 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:06.949356 (+ 559us) server_negotiation.cc:407] Connection header received
0806 02:53:06.949478 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:06.949484 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:06.949583 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:06.949700 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:06.951488 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:06.952591 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:06.954369 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:06.954990 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:06.955119 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:06.958318 (+ 3199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:06.958329 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:06.958347 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:53:06.958392 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:06.961489 (+ 3097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:06.962443 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:06.962452 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:06.962459 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:06.962647 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:06.963079 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:06.963085 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:06.963088 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:06.963616 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:06.963899 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:06.964449 (+ 550us) server_negotiation.cc:299] Negotiation successful
0806 02:53:06.964829 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
Aug 06 02:53:06 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.131@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:53:06 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.131@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:53:06.972661 7779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:06.964190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:35411 (local address 127.25.211.131:60597)
0806 02:53:06.964461 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:06.964481 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:06.964594 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:06.964935 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:06.964940 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:06.965362 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:06.965509 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:06.965538 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:06.966912 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:06.966917 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:06.968142 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:06.968155 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:06.969040 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:06.969046 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:06.969220 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:06.969783 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:06.969800 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:06.972466 (+ 2666us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.211.190:35411: Server oryx/127.25.211.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155}
W20260806 02:53:06.972939 7370 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:53:06.964298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:60597 (local address 127.25.211.190:35411)
0806 02:53:06.964472 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:06.964478 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:06.964532 (+ 54us) server_negotiation.cc:407] Connection header received
0806 02:53:06.964708 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:06.964713 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:06.964787 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:06.964883 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:06.965669 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:06.966766 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:06.968284 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:06.968891 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:06.968983 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:06.972699 (+ 3716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.211.131:60597: BlockingRecv error: recv got EOF from 127.25.211.131:60597 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 06 02:53:06 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.131@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:53:06 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.131@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:53:06.980911 7779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:06.973245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:35411 (local address 127.25.211.131:39345)
0806 02:53:06.973405 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:06.973423 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:06.973515 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:06.973794 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:06.973797 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:06.974053 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:06.974259 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:06.974270 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:06.975192 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:06.975195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:06.976227 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:06.976250 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:06.976847 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:06.976851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:06.976993 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:06.977484 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:06.977500 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:06.980594 (+ 3094us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.211.190:35411: Server oryx/127.25.211.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260806 02:53:06.981114 7370 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:53:06.973343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:39345 (local address 127.25.211.190:35411)
0806 02:53:06.973559 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:06.973563 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:06.973578 (+ 15us) server_negotiation.cc:407] Connection header received
0806 02:53:06.973618 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:06.973622 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:06.973685 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:06.973760 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:06.974387 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:06.975085 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:06.976356 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:06.976742 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:06.976832 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:06.980848 (+ 4016us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.211.131:39345: BlockingRecv error: recv got EOF from 127.25.211.131:39345 (error 108)
Metrics: {"server-negotiator.queue_time_us":121}
W20260806 02:53:06.981196 7775 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 06 02:53:07 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.129@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:53:07 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.129@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:53:07.040412 7781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:07.031785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:35411 (local address 127.25.211.129:60129)
0806 02:53:07.032334 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:07.032354 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:07.032478 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:07.032836 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:07.032839 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:07.033325 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:07.033524 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:07.033534 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:07.034596 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:07.034600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:07.035598 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:07.035612 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:07.036415 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:07.036420 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:07.036571 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:07.037126 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:07.037143 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:07.040165 (+ 3022us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.211.190:35411: Server oryx/127.25.211.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":262,"threads_started":1}
W20260806 02:53:07.040701 7370 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:53:07.031905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:60129 (local address 127.25.211.190:35411)
0806 02:53:07.032127 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:07.032133 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:07.032438 (+ 305us) server_negotiation.cc:407] Connection header received
0806 02:53:07.032642 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:07.032646 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:07.032715 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:07.032809 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:07.033662 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:07.034449 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:07.035760 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:07.036207 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:07.036313 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:07.040449 (+ 4136us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.211.129:60129: BlockingRecv error: recv got EOF from 127.25.211.129:60129 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Aug 06 02:53:07 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.130@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:53:07 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.130@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:53:07.509418 7782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:07.500882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:35411 (local address 127.25.211.130:36613)
0806 02:53:07.501368 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:07.501385 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:07.501501 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:07.501793 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:07.501796 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:07.502324 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:07.502532 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:07.502543 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:07.503566 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:07.503569 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:07.504514 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:07.504526 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:07.505228 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:07.505232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:07.505364 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:07.505962 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:07.505980 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:07.509097 (+ 3117us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.211.190:35411: Server oryx/127.25.211.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":267,"threads_started":1}
W20260806 02:53:07.509580 7370 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:53:07.500990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:36613 (local address 127.25.211.190:35411)
0806 02:53:07.501245 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:07.501252 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:07.501490 (+ 238us) server_negotiation.cc:407] Connection header received
0806 02:53:07.501600 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:07.501604 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:07.501670 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:07.501756 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:07.502680 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:07.503446 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:07.504649 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:07.505083 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:07.505179 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:07.509340 (+ 4161us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.211.130:36613: BlockingRecv error: recv got EOF from 127.25.211.130:36613 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Aug 06 02:53:07 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.131@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:53:07 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.131@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:53:07.991503 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:07.981932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:35411 (local address 127.25.211.131:43255)
0806 02:53:07.982578 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:07.982599 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:07.982792 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:07.983162 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:07.983167 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:07.983799 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:07.984053 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:07.984067 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:07.985173 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:07.985179 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:07.986382 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:07.986398 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:07.987321 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:07.987333 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:07.987528 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:07.988203 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:07.988221 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:07.991263 (+ 3042us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.211.190:35411: Server oryx/127.25.211.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":349,"threads_started":1}
W20260806 02:53:07.991811 7370 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:53:07.982117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:43255 (local address 127.25.211.190:35411)
0806 02:53:07.982396 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:07.982403 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:07.982728 (+ 325us) server_negotiation.cc:407] Connection header received
0806 02:53:07.982916 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:07.982921 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:07.982991 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:07.983088 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:07.984243 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:07.985002 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:07.986539 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:07.987101 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:07.987239 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:07.991541 (+ 4302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.211.131:43255: BlockingRecv error: recv got EOF from 127.25.211.131:43255 (error 108)
Metrics: {"server-negotiator.queue_time_us":114}
Aug 06 02:53:08 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.129@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:53:08 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.129@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:53:08.050289 7788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:08.041708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:35411 (local address 127.25.211.129:39243)
0806 02:53:08.042257 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:08.042274 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:08.042402 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:08.042745 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:08.042749 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:08.043179 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:08.043358 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:08.043368 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:08.044429 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:08.044436 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:08.045362 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:08.045374 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:08.046110 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:08.046115 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:08.046280 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:08.046874 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:08.046891 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:08.049894 (+ 3003us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.211.190:35411: Server oryx/127.25.211.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":282,"threads_started":1}
W20260806 02:53:08.050436 7370 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:53:08.041747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:39243 (local address 127.25.211.190:35411)
0806 02:53:08.041968 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:08.041976 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:08.042357 (+ 381us) server_negotiation.cc:407] Connection header received
0806 02:53:08.042507 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:08.042511 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:08.042574 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:08.042665 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:08.043501 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:08.044278 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:08.045506 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:08.045954 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:08.046069 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:08.050105 (+ 4036us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.211.129:39243: BlockingRecv error: recv got EOF from 127.25.211.129:39243 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 06 02:53:08 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.130@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:53:08 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.130@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:53:08.521261 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:08.510465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:35411 (local address 127.25.211.130:46461)
0806 02:53:08.511008 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:08.511026 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:08.511160 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:08.511474 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:08.511478 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:08.511887 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:08.512063 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:08.512073 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:08.513154 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:08.513158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:08.514080 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:08.514104 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:08.514886 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:08.514891 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:08.515008 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:08.515754 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:08.517480 (+ 1726us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:08.520913 (+ 3433us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.211.190:35411: Server oryx/127.25.211.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":310,"threads_started":1}
W20260806 02:53:08.521410 7370 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:53:08.510580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:46461 (local address 127.25.211.190:35411)
0806 02:53:08.510749 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:08.510754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:08.511115 (+ 361us) server_negotiation.cc:407] Connection header received
0806 02:53:08.511268 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:08.511272 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:08.511341 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:08.511432 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:08.512204 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:08.512977 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:08.514260 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:08.514719 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:08.514821 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:08.521176 (+ 6355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.211.130:46461: BlockingRecv error: recv got EOF from 127.25.211.130:46461 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 06 02:53:08 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.131@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:53:08 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.131@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:53:09.000779 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:08.992502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:35411 (local address 127.25.211.131:50957)
0806 02:53:08.993084 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:08.993099 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:08.993218 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:08.993538 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:08.993542 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:08.993970 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:08.994178 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:08.994192 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:08.995301 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:08.995306 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:08.996222 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:08.996234 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:08.996999 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:08.997003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:08.997130 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:08.997690 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:08.997707 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:09.000522 (+ 2815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.211.190:35411: Server oryx/127.25.211.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":342,"threads_started":1}
W20260806 02:53:09.001096 7370 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:53:08.992727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:50957 (local address 127.25.211.190:35411)
0806 02:53:08.992941 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:08.992948 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:08.993181 (+ 233us) server_negotiation.cc:407] Connection header received
0806 02:53:08.993315 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:08.993319 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:08.993395 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:08.993473 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:08.994348 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:08.995149 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:08.996376 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:08.996836 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:08.996939 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:09.000775 (+ 3836us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.211.131:50957: BlockingRecv error: recv got EOF from 127.25.211.131:50957 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Aug 06 02:53:09 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.129@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:53:09 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.129@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:53:09.059566 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:09.051355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:35411 (local address 127.25.211.129:38365)
0806 02:53:09.051879 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:09.051897 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:09.052054 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:09.052391 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:09.052394 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:09.052807 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:09.052980 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:09.052991 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:09.054020 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:09.054024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:09.055005 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:09.055020 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:09.055718 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:09.055722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:09.055848 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:09.056415 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:09.056432 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:09.059298 (+ 2866us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.211.190:35411: Server oryx/127.25.211.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":272,"threads_started":1}
W20260806 02:53:09.059777 7370 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:53:09.051388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:38365 (local address 127.25.211.190:35411)
0806 02:53:09.051598 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:09.051605 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:09.051990 (+ 385us) server_negotiation.cc:407] Connection header received
0806 02:53:09.052177 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:09.052184 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:09.052258 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:09.052357 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:09.053150 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:09.053883 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:09.055173 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:09.055586 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:09.055693 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:09.059529 (+ 3836us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.211.129:38365: BlockingRecv error: recv got EOF from 127.25.211.129:38365 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 06 02:53:09 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.130@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:53:09 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.130@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:53:09.532799 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:09.522467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:35411 (local address 127.25.211.130:47783)
0806 02:53:09.522957 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:09.522976 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:09.523122 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:09.523748 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:09.523754 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:09.524141 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:09.524347 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:09.524357 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:09.525904 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:09.525919 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:09.527073 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:09.527091 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:09.528064 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:09.528069 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:09.528228 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:09.528899 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:09.528919 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:09.532528 (+ 3609us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.211.190:35411: Server oryx/127.25.211.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":265,"threads_started":1}
W20260806 02:53:09.533113 7370 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:53:09.522648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:47783 (local address 127.25.211.190:35411)
0806 02:53:09.522901 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:09.522908 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:09.523211 (+ 303us) server_negotiation.cc:407] Connection header received
0806 02:53:09.523344 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:09.523351 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:09.523467 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:09.523637 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:09.524528 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:09.525692 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:09.527294 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:09.527864 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:09.527984 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:09.532825 (+ 4841us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.211.130:47783: BlockingRecv error: recv got EOF from 127.25.211.130:47783 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
Aug 06 02:53:10 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.131@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:53:10 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.131@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:53:10.009862 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:10.001653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:35411 (local address 127.25.211.131:45365)
0806 02:53:10.002120 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:10.002136 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:10.002284 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:10.002635 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:10.002639 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:10.002995 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:10.003191 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:10.003201 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:10.004128 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:10.004132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:10.005060 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:10.005074 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:10.005768 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:10.005772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:10.005907 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:10.006538 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:10.006556 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:10.009537 (+ 2981us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.211.190:35411: Server oryx/127.25.211.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":259,"threads_started":1}
W20260806 02:53:10.010085 7370 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:53:10.001781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:45365 (local address 127.25.211.190:35411)
0806 02:53:10.002008 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:10.002014 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:10.002242 (+ 228us) server_negotiation.cc:407] Connection header received
0806 02:53:10.002378 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:10.002382 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:10.002475 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:10.002556 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:10.003313 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:10.004021 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:10.005173 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:10.005614 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:10.005702 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:10.009824 (+ 4122us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.211.131:45365: BlockingRecv error: recv got EOF from 127.25.211.131:45365 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Aug 06 02:53:10 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.129@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:53:10 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.129@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:53:10.069216 7796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:10.060840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:35411 (local address 127.25.211.129:39093)
0806 02:53:10.061371 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:10.061388 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:10.061504 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:10.061880 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:10.061884 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:10.062322 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:10.062483 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:10.062493 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:10.063505 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:10.063509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:10.064530 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:10.064544 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:10.065227 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:10.065231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:10.065359 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:10.065981 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:10.066000 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:10.068951 (+ 2951us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.211.190:35411: Server oryx/127.25.211.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":291,"threads_started":1}
W20260806 02:53:10.069576 7370 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:53:10.060898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:39093 (local address 127.25.211.190:35411)
0806 02:53:10.061132 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:10.061139 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:10.061475 (+ 336us) server_negotiation.cc:407] Connection header received
0806 02:53:10.061642 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:10.061649 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:10.061726 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:10.061805 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:10.062612 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:10.063359 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:10.064655 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:10.065090 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:10.065183 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:10.069195 (+ 4012us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.211.129:39093: BlockingRecv error: recv got EOF from 127.25.211.129:39093 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Aug 06 02:53:10 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.130@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:53:10 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.130@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:53:10.546298 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:10.533818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:35411 (local address 127.25.211.130:49339)
0806 02:53:10.534328 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:10.534346 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:10.534475 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:10.534866 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:10.534870 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:10.535262 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:10.535419 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:10.535428 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:10.536663 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:10.536669 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:10.537996 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:10.538011 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:10.538750 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:10.538754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:10.538901 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:10.543154 (+ 4253us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:10.543174 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:10.545997 (+ 2823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.211.190:35411: Server oryx/127.25.211.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":269,"threads_started":1}
W20260806 02:53:10.546423 7370 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:53:10.533967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:49339 (local address 127.25.211.190:35411)
0806 02:53:10.534218 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:10.534224 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:10.534425 (+ 201us) server_negotiation.cc:407] Connection header received
0806 02:53:10.534631 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:10.534636 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:10.534700 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:10.534781 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:10.535558 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:10.536505 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:10.538184 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:10.538626 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:10.538744 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:10.546226 (+ 7482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.211.130:49339: BlockingRecv error: recv got EOF from 127.25.211.130:49339 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 06 02:53:11 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.131@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:53:11 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.131@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:53:11.019876 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:11.011204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:35411 (local address 127.25.211.131:50939)
0806 02:53:11.011716 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:11.011732 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:11.011852 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:11.012213 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:11.012217 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:11.012657 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:11.012848 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:11.012859 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:11.014115 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:11.014120 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:11.015141 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:11.015155 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:11.015808 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:11.015813 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:11.015946 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:11.016528 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:11.016548 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:11.019630 (+ 3082us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.211.190:35411: Server oryx/127.25.211.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":274,"threads_started":1}
W20260806 02:53:11.020081 7370 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:53:11.011266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:50939 (local address 127.25.211.190:35411)
0806 02:53:11.011434 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:11.011439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:11.011799 (+ 360us) server_negotiation.cc:407] Connection header received
0806 02:53:11.011946 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:11.011950 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:11.012029 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:11.012118 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:11.012984 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:11.013983 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:11.015267 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:11.015660 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:11.015758 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:11.019842 (+ 4084us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.211.131:50939: BlockingRecv error: recv got EOF from 127.25.211.131:50939 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 06 02:53:11 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.129@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:53:11 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.129@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:53:11.078966 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:11.070306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:35411 (local address 127.25.211.129:34791)
0806 02:53:11.070847 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:11.070864 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:11.071003 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:11.071338 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:11.071342 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:11.071809 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:11.071984 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:11.071994 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:11.073053 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:11.073059 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:11.074033 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:11.074051 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:11.075047 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:11.075052 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:11.075209 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:11.075893 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:11.075914 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:11.078706 (+ 2792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.211.190:35411: Server oryx/127.25.211.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":266,"threads_started":1}
W20260806 02:53:11.079118 7370 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:53:11.070376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:34791 (local address 127.25.211.190:35411)
0806 02:53:11.070613 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:11.070621 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:11.070957 (+ 336us) server_negotiation.cc:407] Connection header received
0806 02:53:11.071128 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:11.071135 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:11.071209 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:11.071292 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:11.072153 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:11.072913 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:11.074246 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:11.074865 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:11.074990 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:11.078926 (+ 3936us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.211.129:34791: BlockingRecv error: recv got EOF from 127.25.211.129:34791 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 06 02:53:11 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.130@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:53:11 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.130@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:53:11.556437 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:11.547313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:35411 (local address 127.25.211.130:50527)
0806 02:53:11.547878 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:11.547895 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:11.548012 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:11.548438 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:11.548443 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:11.548893 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:11.549061 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:11.549071 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:11.550343 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:11.550349 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:11.551480 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:11.551496 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:11.552271 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:11.552277 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:11.552439 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:11.553057 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:11.553080 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:11.556164 (+ 3084us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.211.190:35411: Server oryx/127.25.211.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":285,"threads_started":1}
W20260806 02:53:11.556627 7370 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:53:11.547555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:50527 (local address 127.25.211.190:35411)
0806 02:53:11.547733 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:11.547738 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:11.547962 (+ 224us) server_negotiation.cc:407] Connection header received
0806 02:53:11.548180 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:11.548185 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:11.548254 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:11.548357 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:11.549192 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:11.550132 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:11.551624 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:11.552116 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:11.552229 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:11.556395 (+ 4166us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.211.130:50527: BlockingRecv error: recv got EOF from 127.25.211.130:50527 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 06 02:53:12 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.131@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:53:12 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.131@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:53:12.029884 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:12.021052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:35411 (local address 127.25.211.131:36213)
0806 02:53:12.021623 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:12.021643 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:12.021768 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:12.022065 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:12.022068 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:12.022565 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:12.022749 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:12.022759 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:12.023866 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:12.023871 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:12.024987 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:12.025004 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:12.025836 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:12.025842 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:12.026018 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:12.026686 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:12.026708 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:12.029584 (+ 2876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.211.190:35411: Server oryx/127.25.211.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":277,"threads_started":1}
W20260806 02:53:12.030125 7370 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:53:12.021148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:36213 (local address 127.25.211.190:35411)
0806 02:53:12.021378 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:12.021385 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:12.021725 (+ 340us) server_negotiation.cc:407] Connection header received
0806 02:53:12.021863 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:12.021867 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:12.021952 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:12.022036 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:12.022912 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:12.023685 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:12.025129 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:12.025680 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:12.025798 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:12.029837 (+ 4039us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.211.131:36213: BlockingRecv error: recv got EOF from 127.25.211.131:36213 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Aug 06 02:53:12 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.129@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:53:12 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.129@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:53:12.088706 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:12.080015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:35411 (local address 127.25.211.129:60341)
0806 02:53:12.080584 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:12.080600 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:12.080719 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:12.081013 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:12.081016 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:12.081427 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:12.081603 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:12.081613 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:12.082804 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:12.082808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:12.083726 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:12.083741 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:12.084502 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:12.084507 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:12.084656 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:12.085287 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:12.085307 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:12.088468 (+ 3161us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.211.190:35411: Server oryx/127.25.211.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":312,"threads_started":1}
W20260806 02:53:12.088918 7370 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:53:12.080161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:60341 (local address 127.25.211.190:35411)
0806 02:53:12.080347 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:12.080352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:12.080675 (+ 323us) server_negotiation.cc:407] Connection header received
0806 02:53:12.080833 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:12.080837 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:12.080904 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:12.080978 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:12.081770 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:12.082642 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:12.083894 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:12.084327 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:12.084427 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:12.088687 (+ 4260us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.211.129:60341: BlockingRecv error: recv got EOF from 127.25.211.129:60341 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Aug 06 02:53:12 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.130@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:53:12 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.130@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:53:12.566347 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:12.557557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:35411 (local address 127.25.211.130:59739)
0806 02:53:12.558055 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:12.558072 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:12.558240 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:12.558549 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:12.558553 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:12.559059 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:12.559256 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:12.559266 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:12.560535 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:12.560541 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:12.561594 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:12.561608 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:12.562409 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:12.562414 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:12.562549 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:12.563114 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:12.563131 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:12.566008 (+ 2877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.211.190:35411: Server oryx/127.25.211.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":269,"threads_started":1}
W20260806 02:53:12.566583 7370 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:53:12.557717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:59739 (local address 127.25.211.190:35411)
0806 02:53:12.557957 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:12.557963 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:12.558196 (+ 233us) server_negotiation.cc:407] Connection header received
0806 02:53:12.558361 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:12.558366 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:12.558434 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:12.558521 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:12.559402 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:12.560385 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:12.561751 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:12.562235 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:12.562346 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:12.566334 (+ 3988us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.211.130:59739: BlockingRecv error: recv got EOF from 127.25.211.130:59739 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Aug 06 02:53:13 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.131@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:53:13 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.131@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:53:13.040793 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:13.031121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:35411 (local address 127.25.211.131:40009)
0806 02:53:13.031686 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:13.031703 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:13.031838 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:13.032200 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:13.032204 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:13.032618 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:13.032808 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:13.032826 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:13.034096 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:13.034101 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:13.035185 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:13.035200 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:13.036275 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:13.036282 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:13.036461 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:13.037176 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:13.037246 (+ 70us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:13.040461 (+ 3215us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.211.190:35411: Server oryx/127.25.211.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":292,"threads_started":1}
W20260806 02:53:13.040997 7370 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:53:13.031241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:40009 (local address 127.25.211.190:35411)
0806 02:53:13.031444 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:13.031449 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:13.031814 (+ 365us) server_negotiation.cc:407] Connection header received
0806 02:53:13.031944 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:13.031948 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:13.032039 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:13.032134 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:13.032995 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:13.033915 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:13.035417 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:13.036024 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:13.036194 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:13.040752 (+ 4558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.211.131:40009: BlockingRecv error: recv got EOF from 127.25.211.131:40009 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Aug 06 02:53:13 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.129@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:53:13 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.129@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:53:13.099222 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:13.089699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:35411 (local address 127.25.211.129:60805)
0806 02:53:13.090503 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:13.090521 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:13.090693 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:13.091047 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:13.091053 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:13.091503 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:13.091695 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:13.091706 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:13.092757 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:13.092763 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:13.093737 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:13.093752 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:13.094647 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:13.094653 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:13.094815 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:13.095475 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:13.095496 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:13.098842 (+ 3346us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.211.190:35411: Server oryx/127.25.211.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":662,"thread_start_us":333,"threads_started":1}
W20260806 02:53:13.099323 7370 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:53:13.089884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:60805 (local address 127.25.211.190:35411)
0806 02:53:13.090114 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:13.090122 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:13.090609 (+ 487us) server_negotiation.cc:407] Connection header received
0806 02:53:13.090826 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:13.090831 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:13.090898 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:13.090982 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:13.091854 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:13.092606 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:13.093897 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:13.094462 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:13.094580 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:13.099109 (+ 4529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.211.129:60805: BlockingRecv error: recv got EOF from 127.25.211.129:60805 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 06 02:53:13 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.130@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:53:13 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.130@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:53:13.576246 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:13.567439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:35411 (local address 127.25.211.130:35395)
0806 02:53:13.568105 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:13.568123 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:13.568291 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:13.568682 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:13.568687 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:13.569160 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:13.569344 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:13.569354 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:13.570507 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:13.570511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:13.571469 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:13.571482 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:13.572311 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:13.572317 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:13.572482 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:13.573061 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:13.573078 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:13.575981 (+ 2903us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.211.190:35411: Server oryx/127.25.211.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":308,"threads_started":1}
W20260806 02:53:13.576433 7370 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:53:13.567587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:35395 (local address 127.25.211.190:35411)
0806 02:53:13.567844 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:13.567852 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:13.568219 (+ 367us) server_negotiation.cc:407] Connection header received
0806 02:53:13.568422 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:13.568429 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:13.568532 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:13.568620 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:13.569494 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:13.570362 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:13.571617 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:13.572138 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:13.572257 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:13.576184 (+ 3927us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.211.130:35395: BlockingRecv error: recv got EOF from 127.25.211.130:35395 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Aug 06 02:53:14 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.131@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:53:14 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.131@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:53:14.050333 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:14.041792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:35411 (local address 127.25.211.131:55959)
0806 02:53:14.042385 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:14.042404 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:14.042550 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:14.042939 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:14.042945 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:14.043379 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:14.043549 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:14.043559 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:14.044612 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:14.044617 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:14.045551 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:14.045563 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:14.046335 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:14.046341 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:14.046501 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:14.047073 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:14.047092 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:14.049988 (+ 2896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.211.190:35411: Server oryx/127.25.211.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":370,"threads_started":1}
W20260806 02:53:14.050531 7370 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:53:14.041877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:55959 (local address 127.25.211.190:35411)
0806 02:53:14.042255 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:14.042261 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:14.042501 (+ 240us) server_negotiation.cc:407] Connection header received
0806 02:53:14.042708 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:14.042714 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:14.042786 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:14.042869 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:14.043723 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:14.044456 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:14.045702 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:14.046185 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:14.046303 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:14.050301 (+ 3998us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.211.131:55959: BlockingRecv error: recv got EOF from 127.25.211.131:55959 (error 108)
Metrics: {"server-negotiator.queue_time_us":169}
Aug 06 02:53:14 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.129@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:53:14 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.129@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:53:14.109097 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:14.100398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:35411 (local address 127.25.211.129:60677)
0806 02:53:14.100930 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:14.100947 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:14.101225 (+ 278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:14.101567 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:14.101570 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:14.101995 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:14.102206 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:14.102220 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:14.103269 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:14.103273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:14.104221 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:14.104236 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:14.104998 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:14.105004 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:14.105158 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:14.105858 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:14.105878 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:14.108803 (+ 2925us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.211.190:35411: Server oryx/127.25.211.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":280,"threads_started":1}
W20260806 02:53:14.109360 7370 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:53:14.100614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:60677 (local address 127.25.211.190:35411)
0806 02:53:14.100826 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:14.100833 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:14.101057 (+ 224us) server_negotiation.cc:407] Connection header received
0806 02:53:14.101365 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:14.101370 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:14.101438 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:14.101536 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:14.102358 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:14.103108 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:14.104370 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:14.104807 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:14.104921 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:14.109074 (+ 4153us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.211.129:60677: BlockingRecv error: recv got EOF from 127.25.211.129:60677 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 06 02:53:14 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.130@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:53:14 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.130@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:53:14.585877 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:14.577215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:35411 (local address 127.25.211.130:39501)
0806 02:53:14.577784 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:14.577802 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:14.577928 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:14.578314 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:14.578319 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:14.578733 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:14.578898 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:14.578908 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:14.580137 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:14.580141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:14.581109 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:14.581123 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:14.581944 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:14.581949 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:14.582068 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:14.582749 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:14.582768 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:14.585611 (+ 2843us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.211.190:35411: Server oryx/127.25.211.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":293,"threads_started":1}
W20260806 02:53:14.586145 7370 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:53:14.577296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:39501 (local address 127.25.211.190:35411)
0806 02:53:14.577482 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:14.577487 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:14.577924 (+ 437us) server_negotiation.cc:407] Connection header received
0806 02:53:14.578045 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:14.578049 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:14.578124 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:14.578254 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:14.579052 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:14.579967 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:14.581272 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:14.581760 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:14.581868 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:14.585887 (+ 4019us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.211.130:39501: BlockingRecv error: recv got EOF from 127.25.211.130:39501 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 06 02:53:15 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.131@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:53:15 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.131@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:53:15.059710 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:15.051469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:35411 (local address 127.25.211.131:41489)
0806 02:53:15.051985 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:15.052003 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:15.052126 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:15.052475 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:15.052479 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:15.052935 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:15.053101 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:15.053110 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:15.054247 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:15.054251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:15.055215 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:15.055229 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:15.055945 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:15.055949 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:15.056065 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:15.056643 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:15.056661 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:15.059471 (+ 2810us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.211.190:35411: Server oryx/127.25.211.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":277,"threads_started":1}
W20260806 02:53:15.060032 7370 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:53:15.051538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:41489 (local address 127.25.211.190:35411)
0806 02:53:15.051731 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:15.051736 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:15.052123 (+ 387us) server_negotiation.cc:407] Connection header received
0806 02:53:15.052216 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:15.052220 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:15.052300 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:15.052392 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:15.053257 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:15.054055 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:15.055361 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:15.055798 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:15.055892 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:15.059711 (+ 3819us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.211.131:41489: BlockingRecv error: recv got EOF from 127.25.211.131:41489 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 06 02:53:15 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.129@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:53:15 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.129@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:53:15.118808 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:15.110130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:35411 (local address 127.25.211.129:53367)
0806 02:53:15.110699 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:15.110720 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:15.110877 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:15.111244 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:15.111248 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:15.111782 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:15.111992 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:15.112004 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:15.113043 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:15.113048 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:15.114023 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:15.114037 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:15.114771 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:15.114777 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:15.114981 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:15.115549 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:15.115568 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:15.118412 (+ 2844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.211.190:35411: Server oryx/127.25.211.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":277,"threads_started":1}
W20260806 02:53:15.118888 7370 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:53:15.110328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:53367 (local address 127.25.211.190:35411)
0806 02:53:15.110531 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:15.110537 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:15.110811 (+ 274us) server_negotiation.cc:407] Connection header received
0806 02:53:15.111014 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:15.111019 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:15.111098 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:15.111202 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:15.112134 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:15.112894 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:15.114143 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:15.114593 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:15.114683 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:15.118655 (+ 3972us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.211.129:53367: BlockingRecv error: recv got EOF from 127.25.211.129:53367 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Aug 06 02:53:15 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.130@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:53:15 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.130@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:53:15.599617 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:15.586871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:35411 (local address 127.25.211.130:56561)
0806 02:53:15.587420 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:15.587436 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:15.587578 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:15.587937 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:15.587942 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:15.588358 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:15.588528 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:15.588538 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:15.589725 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:15.589730 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:15.590760 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:15.590774 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:15.591610 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:15.591616 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:15.591752 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:15.596188 (+ 4436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:15.596210 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:15.599354 (+ 3144us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.211.190:35411: Server oryx/127.25.211.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":289,"threads_started":1}
W20260806 02:53:15.599850 7370 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:53:15.586992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:56561 (local address 127.25.211.190:35411)
0806 02:53:15.587232 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:15.587239 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:15.587524 (+ 285us) server_negotiation.cc:407] Connection header received
0806 02:53:15.587714 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:15.587721 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:15.587790 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:15.587872 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:15.588704 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:15.589515 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:15.590922 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:15.591394 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:15.591500 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:15.599605 (+ 8105us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.211.130:56561: BlockingRecv error: recv got EOF from 127.25.211.130:56561 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Aug 06 02:53:16 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.131@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:53:16 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.131@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:53:16.069867 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:16.060992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:35411 (local address 127.25.211.131:59197)
0806 02:53:16.061517 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:16.061535 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:16.061678 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:16.062088 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:16.062091 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:16.062533 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:16.062715 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:16.062725 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:16.063823 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:16.063827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:16.064832 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:16.064848 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:16.065739 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:16.065744 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:16.065903 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:16.066611 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:16.066635 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:16.069570 (+ 2935us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.211.190:35411: Server oryx/127.25.211.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":281,"threads_started":1}
W20260806 02:53:16.069995 7370 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:53:16.061110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:59197 (local address 127.25.211.190:35411)
0806 02:53:16.061397 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:16.061403 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:16.061631 (+ 228us) server_negotiation.cc:407] Connection header received
0806 02:53:16.061855 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:16.061860 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:16.061939 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:16.062037 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:16.062886 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:16.063702 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:16.064968 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:16.065588 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:16.065713 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:16.069791 (+ 4078us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.211.131:59197: BlockingRecv error: recv got EOF from 127.25.211.131:59197 (error 108)
Metrics: {"server-negotiator.queue_time_us":127}
Aug 06 02:53:16 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.129@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:53:16 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.129@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:53:16.128615 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:16.119903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:35411 (local address 127.25.211.129:40347)
0806 02:53:16.120473 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:16.120492 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:16.120611 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:16.120899 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:16.120902 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:16.121281 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:16.121433 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:16.121442 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:16.122590 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:16.122595 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:16.123572 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:16.123585 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:16.124374 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:16.124379 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:16.124508 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:16.125100 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:16.125118 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:16.128283 (+ 3165us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.211.190:35411: Server oryx/127.25.211.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":343,"threads_started":1}
W20260806 02:53:16.128980 7370 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:53:16.120020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:40347 (local address 127.25.211.190:35411)
0806 02:53:16.120198 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:16.120203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:16.120592 (+ 389us) server_negotiation.cc:407] Connection header received
0806 02:53:16.120707 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:16.120711 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:16.120777 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:16.120863 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:16.121617 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:16.122467 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:16.123715 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:16.124201 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:16.124305 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:16.128540 (+ 4235us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.211.129:40347: BlockingRecv error: recv got EOF from 127.25.211.129:40347 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 06 02:53:16 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.130@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:53:16 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.130@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:53:16.609237 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:16.600687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:35411 (local address 127.25.211.130:43627)
0806 02:53:16.601186 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:16.601202 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:16.601335 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:16.601682 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:16.601687 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:16.602123 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:16.602319 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:16.602330 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:16.603369 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:16.603373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:16.604329 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:16.604342 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:16.605061 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:16.605067 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:16.605220 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:16.605915 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:16.605933 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:16.608915 (+ 2982us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.211.190:35411: Server oryx/127.25.211.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":272,"threads_started":1}
W20260806 02:53:16.609449 7370 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:53:16.600830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:43627 (local address 127.25.211.190:35411)
0806 02:53:16.600994 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:16.600998 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:16.601306 (+ 308us) server_negotiation.cc:407] Connection header received
0806 02:53:16.601466 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:16.601472 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:16.601539 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:16.601627 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:16.602476 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:16.603226 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:16.604471 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:16.604907 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:16.605018 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:16.609190 (+ 4172us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.211.130:43627: BlockingRecv error: recv got EOF from 127.25.211.130:43627 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 06 02:53:17 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.131@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:53:17 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.131@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:53:17.079689 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:17.070996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:35411 (local address 127.25.211.131:44557)
0806 02:53:17.071749 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:17.071767 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:17.071899 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:17.072230 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:17.072236 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:17.072641 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:17.072803 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:17.072813 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:17.073874 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:17.073878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:17.074881 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:17.074895 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:17.075622 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:17.075628 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:17.075793 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:17.076390 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:17.076407 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:17.079370 (+ 2963us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.211.190:35411: Server oryx/127.25.211.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":515,"threads_started":1}
W20260806 02:53:17.079911 7370 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:53:17.071220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:44557 (local address 127.25.211.190:35411)
0806 02:53:17.071477 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:17.071484 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:17.071860 (+ 376us) server_negotiation.cc:407] Connection header received
0806 02:53:17.072019 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:17.072025 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:17.072091 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:17.072173 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:17.072963 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:17.073704 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:17.075032 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:17.075464 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:17.075578 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:17.079606 (+ 4028us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.211.131:44557: BlockingRecv error: recv got EOF from 127.25.211.131:44557 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Aug 06 02:53:17 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.129@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:53:17 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.129@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:53:17.139055 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:17.129753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:35411 (local address 127.25.211.129:33585)
0806 02:53:17.130344 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:17.130362 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:17.130501 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:17.130858 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:17.130862 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:17.131352 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:17.131600 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:17.131612 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:17.132902 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:17.132908 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:17.133989 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:17.134005 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:17.134825 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:17.134830 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:17.134965 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:17.135598 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:17.135619 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:17.138789 (+ 3170us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.211.190:35411: Server oryx/127.25.211.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":302,"threads_started":1}
W20260806 02:53:17.139300 7370 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:53:17.129911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:33585 (local address 127.25.211.190:35411)
0806 02:53:17.130201 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:17.130207 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:17.130475 (+ 268us) server_negotiation.cc:407] Connection header received
0806 02:53:17.130621 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:17.130626 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:17.130714 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:17.130804 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:17.131768 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:17.132740 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:17.134189 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:17.134643 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:17.134753 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:17.139039 (+ 4286us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.211.129:33585: BlockingRecv error: recv got EOF from 127.25.211.129:33585 (error 108)
Metrics: {"server-negotiator.queue_time_us":114}
Aug 06 02:53:17 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.130@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:53:17 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.130@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:53:17.619036 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:17.610438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:35411 (local address 127.25.211.130:51933)
0806 02:53:17.610998 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:17.611039 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:17.611192 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:17.611538 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:17.611544 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:17.612026 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:17.612221 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:17.612231 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:17.613292 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:17.613296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:17.614258 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:17.614272 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:17.614982 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:17.614988 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:17.615172 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:17.615781 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:17.615799 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:17.618746 (+ 2947us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.211.190:35411: Server oryx/127.25.211.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":323,"threads_started":1}
W20260806 02:53:17.619280 7370 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:53:17.610583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:51933 (local address 127.25.211.190:35411)
0806 02:53:17.610812 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:17.610820 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:17.611128 (+ 308us) server_negotiation.cc:407] Connection header received
0806 02:53:17.611330 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:17.611335 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:17.611400 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:17.611486 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:17.612367 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:17.613150 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:17.614400 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:17.614828 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:17.614939 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:17.618968 (+ 4029us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.211.130:51933: BlockingRecv error: recv got EOF from 127.25.211.130:51933 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Aug 06 02:53:18 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.131@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:53:18 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.131@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:53:18.090296 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:18.080825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:35411 (local address 127.25.211.131:54707)
0806 02:53:18.081590 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:18.081607 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:18.081756 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:18.082197 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:18.082202 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:18.082653 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:18.082846 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:18.082856 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:18.083929 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:18.083933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:18.084840 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:18.084852 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:18.085637 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:18.085643 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:18.085782 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:18.086508 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:18.086527 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:18.089994 (+ 3467us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.211.190:35411: Server oryx/127.25.211.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":262,"threads_started":1}
W20260806 02:53:18.090512 7370 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:53:18.080933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:54707 (local address 127.25.211.190:35411)
0806 02:53:18.081155 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:18.081161 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:18.081713 (+ 552us) server_negotiation.cc:407] Connection header received
0806 02:53:18.081875 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:18.081881 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:18.081961 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:18.082074 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:18.083029 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:18.083806 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:18.084987 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:18.085481 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:18.085609 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:18.090289 (+ 4680us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.211.131:54707: BlockingRecv error: recv got EOF from 127.25.211.131:54707 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 06 02:53:18 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.129@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:53:18 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.129@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:53:18.150041 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:18.140126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:35411 (local address 127.25.211.129:50695)
0806 02:53:18.140706 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:18.140723 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:18.140863 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:18.141267 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:18.141272 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:18.141836 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:18.142026 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:18.142038 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:18.143434 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:18.143440 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:18.144566 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:18.144587 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:18.145738 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:18.145743 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:18.145914 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:18.146713 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:18.146732 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:18.149793 (+ 3061us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.211.190:35411: Server oryx/127.25.211.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":323,"threads_started":1}
W20260806 02:53:18.150413 7370 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:53:18.140256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:50695 (local address 127.25.211.190:35411)
0806 02:53:18.140479 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:18.140486 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:18.140810 (+ 324us) server_negotiation.cc:407] Connection header received
0806 02:53:18.141019 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:18.141049 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:18.141119 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:18.141204 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:18.142226 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:18.143243 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:18.144783 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:18.145552 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:18.145695 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:18.150064 (+ 4369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.211.129:50695: BlockingRecv error: recv got EOF from 127.25.211.129:50695 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 06 02:53:18 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.130@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:53:18 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.130@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:53:18.630051 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:18.620299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:35411 (local address 127.25.211.130:45491)
0806 02:53:18.620864 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:18.620882 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:18.621040 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:18.621396 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:18.621401 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:18.621843 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:18.622004 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:18.622014 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:18.623218 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:18.623224 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:18.624277 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:18.624295 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:18.625030 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:18.625036 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:18.625236 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:18.625858 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:18.625875 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:18.629802 (+ 3927us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.211.190:35411: Server oryx/127.25.211.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":322,"threads_started":1}
W20260806 02:53:18.630402 7370 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:53:18.620394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:45491 (local address 127.25.211.190:35411)
0806 02:53:18.620602 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:18.620609 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:18.620973 (+ 364us) server_negotiation.cc:407] Connection header received
0806 02:53:18.621183 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:18.621188 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:18.621256 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:18.621336 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:18.622183 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:18.623056 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:18.624437 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:18.624876 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:18.624989 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:18.630119 (+ 5130us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.211.130:45491: BlockingRecv error: recv got EOF from 127.25.211.130:45491 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 06 02:53:19 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.131@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:53:19 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.131@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:53:19.100373 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:19.091538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:35411 (local address 127.25.211.131:44387)
0806 02:53:19.092240 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:19.092257 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:19.092410 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:19.092750 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:19.092756 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:19.093210 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:19.093391 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:19.093402 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:19.094510 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:19.094516 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:19.095466 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:19.095480 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:19.096211 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:19.096217 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:19.096411 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:19.097038 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:19.097057 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:19.100013 (+ 2956us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.211.190:35411: Server oryx/127.25.211.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":413,"threads_started":1}
W20260806 02:53:19.100569 7370 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:53:19.091625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:44387 (local address 127.25.211.190:35411)
0806 02:53:19.092024 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:19.092031 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:19.092354 (+ 323us) server_negotiation.cc:407] Connection header received
0806 02:53:19.092532 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:19.092536 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:19.092602 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:19.092686 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:19.093546 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:19.094364 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:19.095619 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:19.096061 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:19.096175 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:19.100313 (+ 4138us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.211.131:44387: BlockingRecv error: recv got EOF from 127.25.211.131:44387 (error 108)
Metrics: {"server-negotiator.queue_time_us":141}
Aug 06 02:53:19 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.129@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:53:19 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.129@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:53:19.160203 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:19.151172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:35411 (local address 127.25.211.129:60279)
0806 02:53:19.151718 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:19.151739 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:19.151897 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:19.152304 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:19.152309 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:19.152828 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:19.153003 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:19.153012 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:19.154120 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:19.154124 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:19.155059 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:19.155072 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:19.155835 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:19.155840 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:19.155981 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:19.156626 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:19.156645 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:19.159807 (+ 3162us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.211.190:35411: Server oryx/127.25.211.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":283,"threads_started":1}
W20260806 02:53:19.160288 7370 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:53:19.151272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:60279 (local address 127.25.211.190:35411)
0806 02:53:19.151462 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:19.151468 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:19.151846 (+ 378us) server_negotiation.cc:407] Connection header received
0806 02:53:19.152024 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:19.152030 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:19.152113 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:19.152202 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:19.153169 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:19.153975 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:19.155213 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:19.155667 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:19.155767 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:19.160083 (+ 4316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.211.129:60279: BlockingRecv error: recv got EOF from 127.25.211.129:60279 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 06 02:53:19 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.130@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:53:19 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.130@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:53:19.639844 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:19.631057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:35411 (local address 127.25.211.130:44895)
0806 02:53:19.631629 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:19.631646 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:19.631766 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:19.632083 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:19.632087 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:19.632495 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:19.632716 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:19.632727 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:19.633878 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:19.633883 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:19.634882 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:19.634896 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:19.635667 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:19.635671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:19.635796 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:19.636417 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:19.636436 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:19.639536 (+ 3100us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.211.190:35411: Server oryx/127.25.211.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":266,"threads_started":1}
W20260806 02:53:19.640079 7370 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:53:19.631186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:44895 (local address 127.25.211.190:35411)
0806 02:53:19.631343 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:19.631348 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:19.631748 (+ 400us) server_negotiation.cc:407] Connection header received
0806 02:53:19.631857 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:19.631862 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:19.631933 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:19.632016 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:19.632867 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:19.633712 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:19.635038 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:19.635463 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:19.635563 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:19.639829 (+ 4266us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.211.130:44895: BlockingRecv error: recv got EOF from 127.25.211.130:44895 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 06 02:53:20 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.131@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:53:20 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.131@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:53:20.110247 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:20.101526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:35411 (local address 127.25.211.131:33881)
0806 02:53:20.102075 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:20.102091 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:20.102243 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:20.102508 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:20.102511 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:20.102906 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:20.103064 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:20.103073 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:20.104142 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:20.104147 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:20.105100 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:20.105113 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:20.105955 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:20.105960 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:20.106188 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:20.106812 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:20.106830 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:20.109830 (+ 3000us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.211.190:35411: Server oryx/127.25.211.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":288,"threads_started":1}
W20260806 02:53:20.110394 7370 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:53:20.101559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:33881 (local address 127.25.211.190:35411)
0806 02:53:20.101718 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:20.101741 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:20.102191 (+ 450us) server_negotiation.cc:407] Connection header received
0806 02:53:20.102334 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:20.102338 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:20.102402 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:20.102482 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:20.103189 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:20.103970 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:20.105261 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:20.105757 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:20.105866 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:20.110112 (+ 4246us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.211.131:33881: BlockingRecv error: recv got EOF from 127.25.211.131:33881 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 06 02:53:20 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.129@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:53:20 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.129@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:53:20.170107 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:20.161421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:35411 (local address 127.25.211.129:42603)
0806 02:53:20.161963 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:20.161983 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:20.162136 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:20.162523 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:20.162528 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:20.163093 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:20.163308 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:20.163320 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:20.164439 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:20.164444 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:20.165438 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:20.165454 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:20.166205 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:20.166212 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:20.166368 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:20.166953 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:20.166971 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:20.169787 (+ 2816us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.211.190:35411: Server oryx/127.25.211.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":280,"threads_started":1}
W20260806 02:53:20.170332 7370 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:53:20.161608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:42603 (local address 127.25.211.190:35411)
0806 02:53:20.161926 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:20.161933 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:20.162064 (+ 131us) server_negotiation.cc:407] Connection header received
0806 02:53:20.162270 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:20.162274 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:20.162340 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:20.162453 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:20.163476 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:20.164282 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:20.165577 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:20.166010 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:20.166104 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:20.170034 (+ 3930us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.211.129:42603: BlockingRecv error: recv got EOF from 127.25.211.129:42603 (error 108)
Metrics: {"server-negotiator.queue_time_us":156}
Aug 06 02:53:20 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.130@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:53:20 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.130@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:53:20.654179 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:20.640966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:35411 (local address 127.25.211.130:39675)
0806 02:53:20.641513 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:20.641530 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:20.641672 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:20.642028 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:20.642033 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:20.642566 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:20.642759 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:20.642769 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:20.643890 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:20.643897 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:20.644850 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:20.644864 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:20.645604 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:20.645611 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:20.645795 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:20.650639 (+ 4844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:20.650666 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:20.653861 (+ 3195us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.211.190:35411: Server oryx/127.25.211.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":287,"threads_started":1}
W20260806 02:53:20.654408 7370 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:53:20.641130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:39675 (local address 127.25.211.190:35411)
0806 02:53:20.641336 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:20.641343 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:20.641617 (+ 274us) server_negotiation.cc:407] Connection header received
0806 02:53:20.641822 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:20.641826 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:20.641890 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:20.641978 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:20.642908 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:20.643743 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:20.644998 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:20.645441 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:20.645580 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:20.654189 (+ 8609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.211.130:39675: BlockingRecv error: recv got EOF from 127.25.211.130:39675 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 06 02:53:21 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.131@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:53:21 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.131@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:53:21.119750 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:21.111324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:35411 (local address 127.25.211.131:36771)
0806 02:53:21.111813 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:21.111828 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:21.111943 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:21.112299 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:21.112304 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:21.112696 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:21.112897 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:21.112908 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:21.113855 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:21.113858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:21.114841 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:21.114853 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:21.115519 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:21.115523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:21.115649 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:21.116257 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:21.116275 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:21.119421 (+ 3146us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.211.190:35411: Server oryx/127.25.211.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":270,"threads_started":1}
W20260806 02:53:21.119915 7370 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:53:21.111386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:36771 (local address 127.25.211.190:35411)
0806 02:53:21.111541 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:21.111546 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:21.111896 (+ 350us) server_negotiation.cc:407] Connection header received
0806 02:53:21.112057 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:21.112062 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:21.112126 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:21.112208 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:21.113015 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:21.113735 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:21.114955 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:21.115398 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:21.115491 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:21.119696 (+ 4205us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.211.131:36771: BlockingRecv error: recv got EOF from 127.25.211.131:36771 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 06 02:53:21 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.129@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:53:21 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.129@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:53:21.179633 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:21.171295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:35411 (local address 127.25.211.129:46907)
0806 02:53:21.171766 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:21.171783 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:21.171897 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:21.172182 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:21.172185 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:21.172597 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:21.172788 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:21.172798 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:21.173766 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:21.173770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:21.174770 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:21.174783 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:21.175443 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:21.175447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:21.175556 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:21.176132 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:21.176150 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:21.179372 (+ 3222us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.211.190:35411: Server oryx/127.25.211.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":243,"threads_started":1}
W20260806 02:53:21.179745 7370 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:53:21.171405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:46907 (local address 127.25.211.190:35411)
0806 02:53:21.171644 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:21.171649 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:21.171852 (+ 203us) server_negotiation.cc:407] Connection header received
0806 02:53:21.171991 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:21.171995 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:21.172060 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:21.172140 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:21.172901 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:21.173639 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:21.174883 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:21.175318 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:21.175412 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:21.179525 (+ 4113us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.211.129:46907: BlockingRecv error: recv got EOF from 127.25.211.129:46907 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Aug 06 02:53:21 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.130@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 02:53:21 dist-test-slave-zj8q krb5kdc[6263](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.25.211.130@KRBTEST.COM for oryx/127.25.211.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 02:53:21.664048 7849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:21.655460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:35411 (local address 127.25.211.130:60027)
0806 02:53:21.656218 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:21.656239 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:21.656371 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:21.656710 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:21.656714 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:21.657172 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:21.657343 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:21.657353 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:21.658474 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:21.658478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:21.659414 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:21.659427 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:21.660185 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:21.660190 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:21.660348 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:21.660935 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:21.660953 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:21.663689 (+ 2736us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.25.211.190:35411: Server oryx/127.25.211.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":282,"threads_started":1}
W20260806 02:53:21.664168 7370 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:53:21.655643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:60027 (local address 127.25.211.190:35411)
0806 02:53:21.655835 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:21.655841 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:21.656329 (+ 488us) server_negotiation.cc:407] Connection header received
0806 02:53:21.656492 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:21.656498 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:21.656565 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:21.656648 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:21.657474 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:21.658307 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:21.659566 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:21.660032 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:21.660141 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:21.663926 (+ 3785us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.25.211.130:60027: BlockingRecv error: recv got EOF from 127.25.211.130:60027 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 02:53:21.952984 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 7172
I20260806 02:53:21.965662 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:21.954141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57248 (local address 127.25.211.129:45679)
0806 02:53:21.954770 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:21.954778 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:21.954805 (+ 27us) server_negotiation.cc:407] Connection header received
0806 02:53:21.954869 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:21.954876 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:21.955093 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:21.955256 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:21.956684 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:21.957772 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:21.959156 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:21.959766 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:21.959872 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:21.962472 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:21.962482 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:21.962499 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:53:21.962537 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:21.965101 (+ 2564us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.25.211.129@KRBTEST.COM not found in keytab (ticket kvno 2)
0806 02:53:21.965391 (+ 290us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57248: Request ticket server kudu/127.25.211.129@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":288,"threads_started":1}
W20260806 02:53:21.965771 7853 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57248: Request ticket server kudu/127.25.211.129@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260806 02:53:21.965883 26446 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 7172: Not authorized: Client connection negotiation failed: client connection to 127.25.211.129:45679: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.25.211.129@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260806 02:53:21.966114 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 7172
I20260806 02:53:21.990476 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 7375
I20260806 02:53:22.003232 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:21.991477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35050 (local address 127.25.211.130:36333)
0806 02:53:21.992067 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:21.992078 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:21.992111 (+ 33us) server_negotiation.cc:407] Connection header received
0806 02:53:21.992179 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:21.992185 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:21.992416 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:21.992572 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:21.993918 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:21.995200 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:21.996537 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:21.997167 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:21.997268 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:21.999856 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:21.999866 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:21.999884 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:53:21.999928 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:22.002701 (+ 2773us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.25.211.130@KRBTEST.COM not found in keytab (ticket kvno 2)
0806 02:53:22.002997 (+ 296us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35050: Request ticket server kudu/127.25.211.130@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":270,"threads_started":1}
W20260806 02:53:22.003365 7854 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35050: Request ticket server kudu/127.25.211.130@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260806 02:53:22.003512 26446 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 7375: Not authorized: Client connection negotiation failed: client connection to 127.25.211.130:36333: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.25.211.130@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260806 02:53:22.003693 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 7375
I20260806 02:53:22.032197 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 7577
I20260806 02:53:22.044432 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:22.033281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51356 (local address 127.25.211.131:39455)
0806 02:53:22.033824 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:22.033832 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:22.033858 (+ 26us) server_negotiation.cc:407] Connection header received
0806 02:53:22.033927 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:22.033934 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:22.034212 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:22.034378 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:22.035761 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:22.036774 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:22.037998 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:22.038718 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:22.038824 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:22.041436 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:22.041445 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:22.041462 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:53:22.041498 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:22.043897 (+ 2399us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.25.211.131@KRBTEST.COM not found in keytab (ticket kvno 2)
0806 02:53:22.044182 (+ 285us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51356: Request ticket server kudu/127.25.211.131@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":261,"threads_started":1}
W20260806 02:53:22.044523 7855 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51356: Request ticket server kudu/127.25.211.131@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260806 02:53:22.044610 26446 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 7577: Not authorized: Client connection negotiation failed: client connection to 127.25.211.131:39455: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.25.211.131@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260806 02:53:22.044785 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 7577
I20260806 02:53:22.066449 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 7062
I20260806 02:53:22.067813 7161 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:53:22.185743 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 7062
2026-08-06T02:53:22Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21725 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-06T02:53:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:53:22Z Disabled control of system clock
I20260806 02:53:22.281415 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:40331
--webserver_interface=127.25.211.190
--webserver_port=0
--builtin_ntp_servers=127.25.211.148:34185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.211.190:40331
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260806 02:53:22.636317 7862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:53:22.636643 7862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:53:22.636736 7862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:53:22.644259 7862 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:53:22.644394 7862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:53:22.644433 7862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:53:22.644492 7862 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:53:22.644537 7862 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:53:22.654528 7862 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:34185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.211.190:40331
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:40331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.211.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:53:22.655846 7862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:53:22.657632 7862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:53:22.670020 7876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:22.670056 7873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:22.670020 7871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:53:22.670578 7862 server_base.cc:1034] running on GCE node
I20260806 02:53:22.671590 7862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:53:22.673101 7862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:53:22.674307 7862 hybrid_clock.cc:648] HybridClock initialized: now 1785984802674254 us; error 42 us; skew 500 ppm
I20260806 02:53:22.674674 7862 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:53:22.677042 7862 webserver.cc:460] Webserver started at http://127.25.211.190:38623/ using document root <none> and password file <none>
I20260806 02:53:22.677815 7862 fs_manager.cc:359] Metadata directory not provided
I20260806 02:53:22.677930 7862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:53:22.678310 7862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:53:22.681385 7862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/master-0/data/instance:
uuid: "eb9b8f9bab8740c8818c1072a6f2a728"
format_stamp: "Formatted at 2026-08-06 02:53:22 on dist-test-slave-zj8q"
server_key: "605067f308bcd684c056c6165417821f"
server_key_iv: "4daca26ea01d301d661161734187cc54"
server_key_version: "encryptionkey@0"
I20260806 02:53:22.682230 7862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/master-0/wal/instance:
uuid: "eb9b8f9bab8740c8818c1072a6f2a728"
format_stamp: "Formatted at 2026-08-06 02:53:22 on dist-test-slave-zj8q"
server_key: "605067f308bcd684c056c6165417821f"
server_key_iv: "4daca26ea01d301d661161734187cc54"
server_key_version: "encryptionkey@0"
I20260806 02:53:22.687556 7862 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260806 02:53:22.691084 7883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:22.692538 7862 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260806 02:53:22.692759 7862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/master-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/master-0/wal
uuid: "eb9b8f9bab8740c8818c1072a6f2a728"
format_stamp: "Formatted at 2026-08-06 02:53:22 on dist-test-slave-zj8q"
server_key: "605067f308bcd684c056c6165417821f"
server_key_iv: "4daca26ea01d301d661161734187cc54"
server_key_version: "encryptionkey@0"
I20260806 02:53:22.692996 7862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:53:22.710587 7862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:53:22.711652 7862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:53:22.711951 7862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:53:22.730381 7862 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.190:40331
I20260806 02:53:22.730389 7954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.190:40331 every 8 connection(s)
I20260806 02:53:22.731714 7862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
I20260806 02:53:22.735483 7957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:53:22.742266 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 7862
I20260806 02:53:22.742457 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/master-0/wal/instance
I20260806 02:53:22.743124 26446 external_mini_cluster.cc:1428] Setting key 4a7a4dd92296fcaeea7cec3c7e3da835
I20260806 02:53:22.746196 7957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb9b8f9bab8740c8818c1072a6f2a728: Bootstrap starting.
I20260806 02:53:22.750494 7957 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb9b8f9bab8740c8818c1072a6f2a728: Neither blocks nor log segments found. Creating new log.
I20260806 02:53:22.751943 7957 log.cc:824] T 00000000000000000000000000000000 P eb9b8f9bab8740c8818c1072a6f2a728: Log is configured to *not* fsync() on all Append() calls
I20260806 02:53:22.755088 7957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb9b8f9bab8740c8818c1072a6f2a728: No bootstrap required, opened a new log
I20260806 02:53:22.758895 7960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:22.745170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34834 (local address 127.25.211.190:40331)
0806 02:53:22.746329 (+ 1159us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:22.746345 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:22.746398 (+ 53us) server_negotiation.cc:407] Connection header received
0806 02:53:22.748907 (+ 2509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:22.748938 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:22.749560 (+ 622us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:22.749993 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:22.750792 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:22.752834 (+ 2042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:22.754256 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:22.755007 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:22.755192 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:22.756348 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:22.756356 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:22.756371 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:53:22.756969 (+ 598us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:22.757069 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:53:22.757093 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:22.757205 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:22.757464 (+ 259us) server_negotiation.cc:299] Negotiation successful
0806 02:53:22.757788 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":353,"threads_started":1}
I20260806 02:53:22.763727 7957 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb9b8f9bab8740c8818c1072a6f2a728 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9b8f9bab8740c8818c1072a6f2a728" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 40331 } }
I20260806 02:53:22.764156 7957 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb9b8f9bab8740c8818c1072a6f2a728 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:53:22.764252 7957 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb9b8f9bab8740c8818c1072a6f2a728 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb9b8f9bab8740c8818c1072a6f2a728, State: Initialized, Role: FOLLOWER
I20260806 02:53:22.764814 7957 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb9b8f9bab8740c8818c1072a6f2a728 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9b8f9bab8740c8818c1072a6f2a728" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 40331 } }
I20260806 02:53:22.765045 7957 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb9b8f9bab8740c8818c1072a6f2a728 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:53:22.765177 7957 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb9b8f9bab8740c8818c1072a6f2a728 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:53:22.765341 7957 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb9b8f9bab8740c8818c1072a6f2a728 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:53:22.767956 7957 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb9b8f9bab8740c8818c1072a6f2a728 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9b8f9bab8740c8818c1072a6f2a728" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 40331 } }
I20260806 02:53:22.768499 7957 leader_election.cc:302] T 00000000000000000000000000000000 P eb9b8f9bab8740c8818c1072a6f2a728 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb9b8f9bab8740c8818c1072a6f2a728; no voters:
I20260806 02:53:22.769094 7957 leader_election.cc:288] T 00000000000000000000000000000000 P eb9b8f9bab8740c8818c1072a6f2a728 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:53:22.769253 7962 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb9b8f9bab8740c8818c1072a6f2a728 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:53:22.770015 7962 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb9b8f9bab8740c8818c1072a6f2a728 [term 1 LEADER]: Becoming Leader. State: Replica: eb9b8f9bab8740c8818c1072a6f2a728, State: Running, Role: LEADER
I20260806 02:53:22.770628 7962 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb9b8f9bab8740c8818c1072a6f2a728 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9b8f9bab8740c8818c1072a6f2a728" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 40331 } }
I20260806 02:53:22.771214 7957 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb9b8f9bab8740c8818c1072a6f2a728 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:53:22.774744 7965 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb9b8f9bab8740c8818c1072a6f2a728 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb9b8f9bab8740c8818c1072a6f2a728. Latest consensus state: current_term: 1 leader_uuid: "eb9b8f9bab8740c8818c1072a6f2a728" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9b8f9bab8740c8818c1072a6f2a728" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 40331 } } }
I20260806 02:53:22.775048 7965 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb9b8f9bab8740c8818c1072a6f2a728 [sys.catalog]: This master's current role is: LEADER
I20260806 02:53:22.775244 7964 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb9b8f9bab8740c8818c1072a6f2a728 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb9b8f9bab8740c8818c1072a6f2a728" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb9b8f9bab8740c8818c1072a6f2a728" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 40331 } } }
I20260806 02:53:22.775718 7964 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb9b8f9bab8740c8818c1072a6f2a728 [sys.catalog]: This master's current role is: LEADER
I20260806 02:53:22.775985 7977 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:53:22.781920 7977 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:53:22.791181 7977 catalog_manager.cc:1347] Generated new cluster ID: df259da6f18945c1a731fe76c30b03ea
I20260806 02:53:22.791327 7977 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:53:22.804809 7977 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:53:22.806905 7977 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:53:22.821063 7977 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb9b8f9bab8740c8818c1072a6f2a728: Generated new TSK 0
I20260806 02:53:22.822077 7977 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:53:22.833956 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.129:0
--local_ip_for_outbound_sockets=127.25.211.129
--webserver_interface=127.25.211.129
--webserver_port=0
--tserver_master_addrs=127.25.211.190:40331
--builtin_ntp_servers=127.25.211.148:34185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260806 02:53:23.206238 7991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:53:23.206581 7991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:53:23.206709 7991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:53:23.214258 7991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:53:23.214388 7991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:53:23.214545 7991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.129
I20260806 02:53:23.224018 7991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:34185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.211.129
--webserver_port=0
--tserver_master_addrs=127.25.211.190:40331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.129
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:53:23.225376 7991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:53:23.227222 7991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:53:23.240499 7999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:23.240770 8001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:23.240617 7998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:53:23.241267 7991 server_base.cc:1034] running on GCE node
I20260806 02:53:23.242046 7991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:53:23.243237 7991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:53:23.244467 7991 hybrid_clock.cc:648] HybridClock initialized: now 1785984803244399 us; error 65 us; skew 500 ppm
I20260806 02:53:23.244874 7991 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:53:23.247452 7991 webserver.cc:460] Webserver started at http://127.25.211.129:37541/ using document root <none> and password file <none>
I20260806 02:53:23.248245 7991 fs_manager.cc:359] Metadata directory not provided
I20260806 02:53:23.248361 7991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:53:23.248730 7991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:53:23.251991 7991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-0/data/instance:
uuid: "4ba8fbd83f8345df8700efd0b25e6a11"
format_stamp: "Formatted at 2026-08-06 02:53:23 on dist-test-slave-zj8q"
server_key: "02564eacf730af9aa38201541c1e7252"
server_key_iv: "64380794273111bbc7f78429b9929d44"
server_key_version: "encryptionkey@0"
I20260806 02:53:23.253089 7991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-0/wal/instance:
uuid: "4ba8fbd83f8345df8700efd0b25e6a11"
format_stamp: "Formatted at 2026-08-06 02:53:23 on dist-test-slave-zj8q"
server_key: "02564eacf730af9aa38201541c1e7252"
server_key_iv: "64380794273111bbc7f78429b9929d44"
server_key_version: "encryptionkey@0"
I20260806 02:53:23.259301 7991 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260806 02:53:23.263373 8008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:23.264972 7991 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:53:23.265168 7991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-0/wal
uuid: "4ba8fbd83f8345df8700efd0b25e6a11"
format_stamp: "Formatted at 2026-08-06 02:53:23 on dist-test-slave-zj8q"
server_key: "02564eacf730af9aa38201541c1e7252"
server_key_iv: "64380794273111bbc7f78429b9929d44"
server_key_version: "encryptionkey@0"
I20260806 02:53:23.265409 7991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:53:23.280669 7991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:53:23.281565 7991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:53:23.281870 7991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:53:23.282948 7991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:53:23.284467 7991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:53:23.284574 7991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:23.284685 7991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:53:23.284762 7991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:23.319159 7991 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.129:43937
I20260806 02:53:23.319201 8183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.129:43937 every 8 connection(s)
I20260806 02:53:23.320761 7991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
I20260806 02:53:23.330315 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 7991
I20260806 02:53:23.330530 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-0/wal/instance
I20260806 02:53:23.331319 26446 external_mini_cluster.cc:1428] Setting key 287c6486dd1a85b089a82b7e36345878
I20260806 02:53:23.334406 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.130:0
--local_ip_for_outbound_sockets=127.25.211.130
--webserver_interface=127.25.211.130
--webserver_port=0
--tserver_master_addrs=127.25.211.190:40331
--builtin_ntp_servers=127.25.211.148:34185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260806 02:53:23.334664 8189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:23.323457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:37831 (local address 127.25.211.190:40331)
0806 02:53:23.323932 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:23.323938 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:23.325115 (+ 1177us) server_negotiation.cc:407] Connection header received
0806 02:53:23.327007 (+ 1892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:23.327012 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:23.327120 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:23.327265 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:23.328420 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:23.329172 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:23.331531 (+ 2359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:23.332007 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:23.332160 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:23.333720 (+ 1560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:23.333728 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:23.333732 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:53:23.333754 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:23.333804 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:53:23.333820 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:23.333911 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:23.334183 (+ 272us) server_negotiation.cc:299] Negotiation successful
0806 02:53:23.334297 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":254,"threads_started":1}
I20260806 02:53:23.335690 8191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:23.323955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:40331 (local address 127.25.211.129:37831)
0806 02:53:23.324946 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:23.324983 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:23.326717 (+ 1734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:23.327518 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:23.327527 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:23.327631 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:23.328242 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:23.328261 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:23.329356 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:23.329360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:23.331325 (+ 1965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:23.331344 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:23.332257 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:23.332263 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:23.332573 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:23.333368 (+ 795us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:53:23.333371 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:23.333416 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:53:23.333508 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:53:23.333928 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:53:23.333938 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:53:23.333945 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 02:53:23.334118 (+ 173us) client_negotiation.cc:240] Negotiation successful
0806 02:53:23.334506 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":302,"threads_started":1}
I20260806 02:53:23.337218 8184 heartbeater.cc:347] Connected to a master server at 127.25.211.190:40331
I20260806 02:53:23.337514 8184 heartbeater.cc:464] Registering TS with master...
I20260806 02:53:23.338362 8184 heartbeater.cc:511] Master 127.25.211.190:40331 requested a full tablet report, sending...
I20260806 02:53:23.340574 7907 ts_manager.cc:194] Registered new tserver with Master: 4ba8fbd83f8345df8700efd0b25e6a11 (127.25.211.129:43937)
I20260806 02:53:23.342082 7907 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.211.129:37831
W20260806 02:53:23.680428 8192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:53:23.680739 8192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:53:23.680822 8192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:53:23.688199 8192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:53:23.688325 8192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:53:23.688431 8192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.130
I20260806 02:53:23.697856 8192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:34185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.211.130
--webserver_port=0
--tserver_master_addrs=127.25.211.190:40331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.130
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:53:23.699214 8192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:53:23.701018 8192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:53:23.714636 8202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:23.714699 8208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:23.714843 8203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:53:23.715422 8192 server_base.cc:1034] running on GCE node
I20260806 02:53:23.716049 8192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:53:23.717037 8192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:53:23.718266 8192 hybrid_clock.cc:648] HybridClock initialized: now 1785984803718205 us; error 51 us; skew 500 ppm
I20260806 02:53:23.718586 8192 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:53:23.720897 8192 webserver.cc:460] Webserver started at http://127.25.211.130:38841/ using document root <none> and password file <none>
I20260806 02:53:23.721616 8192 fs_manager.cc:359] Metadata directory not provided
I20260806 02:53:23.721719 8192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:53:23.722039 8192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:53:23.725063 8192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-1/data/instance:
uuid: "974ecf32d9df44d48cd58d3ae3b92241"
format_stamp: "Formatted at 2026-08-06 02:53:23 on dist-test-slave-zj8q"
server_key: "1527652621c59a3e5fd9cf801c3e0b29"
server_key_iv: "1a13deb41e91eef0526c6e24799d988b"
server_key_version: "encryptionkey@0"
I20260806 02:53:23.725914 8192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-1/wal/instance:
uuid: "974ecf32d9df44d48cd58d3ae3b92241"
format_stamp: "Formatted at 2026-08-06 02:53:23 on dist-test-slave-zj8q"
server_key: "1527652621c59a3e5fd9cf801c3e0b29"
server_key_iv: "1a13deb41e91eef0526c6e24799d988b"
server_key_version: "encryptionkey@0"
I20260806 02:53:23.731503 8192 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260806 02:53:23.735284 8216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:23.736811 8192 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:53:23.736986 8192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-1/wal
uuid: "974ecf32d9df44d48cd58d3ae3b92241"
format_stamp: "Formatted at 2026-08-06 02:53:23 on dist-test-slave-zj8q"
server_key: "1527652621c59a3e5fd9cf801c3e0b29"
server_key_iv: "1a13deb41e91eef0526c6e24799d988b"
server_key_version: "encryptionkey@0"
I20260806 02:53:23.737217 8192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:53:23.756083 8192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:53:23.756968 8192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:53:23.757261 8192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:53:23.758283 8192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:53:23.759797 8192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:53:23.759897 8192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:23.760010 8192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:53:23.760078 8192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:23.792524 8192 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.130:46035
I20260806 02:53:23.792567 8388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.130:46035 every 8 connection(s)
I20260806 02:53:23.793912 8192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
I20260806 02:53:23.794562 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 8192
I20260806 02:53:23.794726 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-1/wal/instance
I20260806 02:53:23.795312 26446 external_mini_cluster.cc:1428] Setting key 3f0d4f0c0befb01475f3e5aa36142103
I20260806 02:53:23.798730 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.131:0
--local_ip_for_outbound_sockets=127.25.211.131
--webserver_interface=127.25.211.131
--webserver_port=0
--tserver_master_addrs=127.25.211.190:40331
--builtin_ntp_servers=127.25.211.148:34185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260806 02:53:23.811074 8189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:23.796656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:58341 (local address 127.25.211.190:40331)
0806 02:53:23.796851 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:23.796857 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:23.798895 (+ 2038us) server_negotiation.cc:407] Connection header received
0806 02:53:23.801385 (+ 2490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:23.801390 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:23.801468 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:23.801559 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:23.803137 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:23.803905 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:23.806527 (+ 2622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:23.806991 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:23.807112 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:23.809700 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:23.809706 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:23.809709 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:53:23.809733 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:23.809780 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:53:23.809793 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:23.809873 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:23.810515 (+ 642us) server_negotiation.cc:299] Negotiation successful
0806 02:53:23.810687 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 02:53:23.812407 8391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:23.797440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:40331 (local address 127.25.211.130:58341)
0806 02:53:23.798683 (+ 1243us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:23.798737 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:23.801026 (+ 2289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:23.801872 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:23.801884 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:23.802016 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:23.802898 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:23.802925 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:23.804086 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:23.804091 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:23.806237 (+ 2146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:23.806265 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:23.807665 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:23.807683 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:23.808045 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:23.809112 (+ 1067us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:53:23.809117 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:23.809203 (+ 86us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:53:23.809339 (+ 136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:53:23.809976 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:53:23.809989 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:53:23.809999 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 02:53:23.810325 (+ 326us) client_negotiation.cc:240] Negotiation successful
0806 02:53:23.810865 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":357,"threads_started":1}
I20260806 02:53:23.814220 8389 heartbeater.cc:347] Connected to a master server at 127.25.211.190:40331
I20260806 02:53:23.814990 8389 heartbeater.cc:464] Registering TS with master...
I20260806 02:53:23.815989 8389 heartbeater.cc:511] Master 127.25.211.190:40331 requested a full tablet report, sending...
I20260806 02:53:23.818217 7907 ts_manager.cc:194] Registered new tserver with Master: 974ecf32d9df44d48cd58d3ae3b92241 (127.25.211.130:46035)
I20260806 02:53:23.819373 7907 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.211.130:58341
W20260806 02:53:24.151156 8393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:53:24.151476 8393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:53:24.151609 8393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:53:24.159003 8393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:53:24.159118 8393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:53:24.159264 8393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.131
I20260806 02:53:24.168606 8393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:34185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.211.131
--webserver_port=0
--tserver_master_addrs=127.25.211.190:40331
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.131
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:53:24.170038 8393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:53:24.172011 8393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:53:24.185214 8400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:24.185253 8404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:24.185164 8401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:53:24.186177 8393 server_base.cc:1034] running on GCE node
I20260806 02:53:24.186818 8393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:53:24.187717 8393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:53:24.188907 8393 hybrid_clock.cc:648] HybridClock initialized: now 1785984804188843 us; error 55 us; skew 500 ppm
I20260806 02:53:24.189203 8393 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:53:24.191407 8393 webserver.cc:460] Webserver started at http://127.25.211.131:38729/ using document root <none> and password file <none>
I20260806 02:53:24.192135 8393 fs_manager.cc:359] Metadata directory not provided
I20260806 02:53:24.192243 8393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:53:24.192555 8393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:53:24.195420 8393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-2/data/instance:
uuid: "6d74dc2a0f724ea58ed74be31f1924e9"
format_stamp: "Formatted at 2026-08-06 02:53:24 on dist-test-slave-zj8q"
server_key: "d8e97bb3f9d096299dbe275394e7ad36"
server_key_iv: "4272fa2db09bd5b38ab99ece2aca45e3"
server_key_version: "encryptionkey@0"
I20260806 02:53:24.196172 8393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-2/wal/instance:
uuid: "6d74dc2a0f724ea58ed74be31f1924e9"
format_stamp: "Formatted at 2026-08-06 02:53:24 on dist-test-slave-zj8q"
server_key: "d8e97bb3f9d096299dbe275394e7ad36"
server_key_iv: "4272fa2db09bd5b38ab99ece2aca45e3"
server_key_version: "encryptionkey@0"
I20260806 02:53:24.201244 8393 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260806 02:53:24.204695 8413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:24.206116 8393 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:53:24.206341 8393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-2/wal
uuid: "6d74dc2a0f724ea58ed74be31f1924e9"
format_stamp: "Formatted at 2026-08-06 02:53:24 on dist-test-slave-zj8q"
server_key: "d8e97bb3f9d096299dbe275394e7ad36"
server_key_iv: "4272fa2db09bd5b38ab99ece2aca45e3"
server_key_version: "encryptionkey@0"
I20260806 02:53:24.206562 8393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:53:24.220604 8393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:53:24.221426 8393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:53:24.221726 8393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:53:24.222716 8393 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:53:24.224143 8393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:53:24.224241 8393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:24.224346 8393 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:53:24.224413 8393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:24.256832 8393 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.131:44719
I20260806 02:53:24.256886 8586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.131:44719 every 8 connection(s)
I20260806 02:53:24.258211 8393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
I20260806 02:53:24.263132 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 8393
I20260806 02:53:24.263317 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-2/wal/instance
I20260806 02:53:24.263867 26446 external_mini_cluster.cc:1428] Setting key f2c35199d3fabc03b7940d79becd871c
I20260806 02:53:24.273032 8189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:24.260869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:43843 (local address 127.25.211.190:40331)
0806 02:53:24.261076 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:24.261080 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:24.262613 (+ 1533us) server_negotiation.cc:407] Connection header received
0806 02:53:24.265097 (+ 2484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:24.265104 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:24.265203 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:24.265311 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:24.266877 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:24.267912 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:24.269804 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:24.270377 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:24.270491 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:24.272076 (+ 1585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:24.272081 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:24.272084 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:53:24.272107 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:24.272141 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:53:24.272154 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:24.272234 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:24.272580 (+ 346us) server_negotiation.cc:299] Negotiation successful
0806 02:53:24.272719 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 02:53:24.274403 8589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:24.261383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:40331 (local address 127.25.211.131:43843)
0806 02:53:24.262433 (+ 1050us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:24.262481 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:24.264738 (+ 2257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:24.265682 (+ 944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:24.265698 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:24.265842 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:24.266693 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:24.266716 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:24.268129 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:24.268134 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:24.269607 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:24.269625 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:24.270632 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:24.270638 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:24.270960 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:24.271681 (+ 721us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:53:24.271683 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:24.271732 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:53:24.271852 (+ 120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:53:24.272294 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:53:24.272304 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:53:24.272311 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 02:53:24.272518 (+ 207us) client_negotiation.cc:240] Negotiation successful
0806 02:53:24.272921 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":326,"threads_started":1}
I20260806 02:53:24.277212 8587 heartbeater.cc:347] Connected to a master server at 127.25.211.190:40331
I20260806 02:53:24.278038 8587 heartbeater.cc:464] Registering TS with master...
I20260806 02:53:24.279274 8587 heartbeater.cc:511] Master 127.25.211.190:40331 requested a full tablet report, sending...
I20260806 02:53:24.280869 7907 ts_manager.cc:194] Registered new tserver with Master: 6d74dc2a0f724ea58ed74be31f1924e9 (127.25.211.131:44719)
I20260806 02:53:24.281749 7907 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.211.131:43843
I20260806 02:53:24.293249 26446 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 02:53:24.299486 8189 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:53:24.297940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34836 (local address 127.25.211.190:40331)
0806 02:53:24.298109 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:24.298113 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:24.298588 (+ 475us) server_negotiation.cc:407] Connection header received
0806 02:53:24.298743 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:24.298747 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:24.298823 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:24.298911 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:24.299252 (+ 341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34836: BlockingRecv error: recv got EOF from 127.0.0.1:34836 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 02:53:24.300283 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 7991
I20260806 02:53:24.308255 8598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:24.301252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56874 (local address 127.25.211.129:43937)
0806 02:53:24.301784 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:24.301791 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:24.301820 (+ 29us) server_negotiation.cc:407] Connection header received
0806 02:53:24.301888 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:24.301894 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:24.302119 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:24.302309 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:24.302717 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:24.303799 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:24.305117 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:24.305734 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:24.305846 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:24.306892 (+ 1046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:24.306900 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:24.306915 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:53:24.307541 (+ 626us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:24.307593 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:53:24.307621 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:24.307720 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:24.307816 (+ 96us) server_negotiation.cc:299] Negotiation successful
0806 02:53:24.307952 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":283,"threads_started":1}
I20260806 02:53:24.309674 8178 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:53:24.397498 8184 heartbeater.cc:503] Master 127.25.211.190:40331 was elected leader, sending a full tablet report...
I20260806 02:53:24.398999 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 7991
W20260806 02:53:24.429342 7884 connection.cc:535] server connection from 127.25.211.129:37831 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 02:53:24.429616 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 8192
I20260806 02:53:24.437987 8601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:24.430515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50504 (local address 127.25.211.130:46035)
0806 02:53:24.431068 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:24.431076 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:24.431113 (+ 37us) server_negotiation.cc:407] Connection header received
0806 02:53:24.431173 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:24.431180 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:24.431407 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:24.431525 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:24.431936 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:24.433003 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:24.434591 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:24.435202 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:24.435343 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:24.436428 (+ 1085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:24.436437 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:24.436453 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:53:24.437150 (+ 697us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:24.437201 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:53:24.437223 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:24.437311 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:24.437454 (+ 143us) server_negotiation.cc:299] Negotiation successful
0806 02:53:24.437642 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":299,"threads_started":1}
I20260806 02:53:24.439324 8380 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:53:24.521443 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 8192
I20260806 02:53:24.541200 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 8393
I20260806 02:53:24.549363 8603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:24.542306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42728 (local address 127.25.211.131:44719)
0806 02:53:24.542957 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:24.542964 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:24.542995 (+ 31us) server_negotiation.cc:407] Connection header received
0806 02:53:24.543062 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:24.543068 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:24.543281 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:24.543435 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:24.543901 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:24.544960 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:24.546271 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:24.546869 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:24.546980 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:24.547965 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:24.547975 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:24.547989 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:53:24.548607 (+ 618us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:24.548661 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:53:24.548697 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:24.548789 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:24.548885 (+ 96us) server_negotiation.cc:299] Negotiation successful
0806 02:53:24.549031 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":330,"threads_started":1}
I20260806 02:53:24.550683 8581 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:53:24.633335 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 8393
I20260806 02:53:24.653405 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 7862
I20260806 02:53:24.654650 7947 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:53:24.760201 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 7862
2026-08-06T02:53:24Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2571 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-06T02:53:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:53:24Z Disabled control of system clock
I20260806 02:53:24.855511 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:37613
--webserver_interface=127.25.211.190
--webserver_port=0
--builtin_ntp_servers=127.25.211.148:42299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.211.190:37613
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260806 02:53:25.211516 8613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:53:25.211892 8613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:53:25.212031 8613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:53:25.219514 8613 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:53:25.219661 8613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:53:25.219732 8613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:53:25.219827 8613 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:53:25.219890 8613 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:53:25.229609 8613 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:42299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.211.190:37613
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:37613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.211.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:53:25.231122 8613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:53:25.233125 8613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:53:25.245776 8621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:25.245836 8622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:25.245852 8624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:53:25.247493 8613 server_base.cc:1034] running on GCE node
I20260806 02:53:25.248394 8613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:53:25.249971 8613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:53:25.251194 8613 hybrid_clock.cc:648] HybridClock initialized: now 1785984805251130 us; error 58 us; skew 500 ppm
I20260806 02:53:25.251564 8613 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:53:25.253976 8613 webserver.cc:460] Webserver started at http://127.25.211.190:33251/ using document root <none> and password file <none>
I20260806 02:53:25.254823 8613 fs_manager.cc:359] Metadata directory not provided
I20260806 02:53:25.254930 8613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:53:25.255266 8613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:53:25.258550 8613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/master-0/data/instance:
uuid: "3a1b303caff746a1a0f3617c977a2156"
format_stamp: "Formatted at 2026-08-06 02:53:25 on dist-test-slave-zj8q"
server_key: "e03d2ecfd30b104aab01576dd9bb3249"
server_key_iv: "bb218597ae92da65ad266e3565875f80"
server_key_version: "encryptionkey@0"
I20260806 02:53:25.259472 8613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/master-0/wal/instance:
uuid: "3a1b303caff746a1a0f3617c977a2156"
format_stamp: "Formatted at 2026-08-06 02:53:25 on dist-test-slave-zj8q"
server_key: "e03d2ecfd30b104aab01576dd9bb3249"
server_key_iv: "bb218597ae92da65ad266e3565875f80"
server_key_version: "encryptionkey@0"
I20260806 02:53:25.265287 8613 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 02:53:25.269495 8631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:25.271432 8613 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 02:53:25.271677 8613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/master-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/master-0/wal
uuid: "3a1b303caff746a1a0f3617c977a2156"
format_stamp: "Formatted at 2026-08-06 02:53:25 on dist-test-slave-zj8q"
server_key: "e03d2ecfd30b104aab01576dd9bb3249"
server_key_iv: "bb218597ae92da65ad266e3565875f80"
server_key_version: "encryptionkey@0"
I20260806 02:53:25.271930 8613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:53:25.278504 8613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:53:25.279414 8613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:53:25.279744 8613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:53:25.299224 8613 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.190:37613
I20260806 02:53:25.299254 8714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.190:37613 every 8 connection(s)
I20260806 02:53:25.300717 8613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
I20260806 02:53:25.305091 8716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:53:25.306123 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 8613
I20260806 02:53:25.306385 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/master-0/wal/instance
I20260806 02:53:25.307235 26446 external_mini_cluster.cc:1428] Setting key ca1704e5f9213a60812b7d47f3911863
I20260806 02:53:25.316761 8722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:25.309630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59996 (local address 127.25.211.190:37613)
0806 02:53:25.310837 (+ 1207us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:25.310850 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:25.310899 (+ 49us) server_negotiation.cc:407] Connection header received
0806 02:53:25.312512 (+ 1613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:25.312547 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:25.313040 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:25.313627 (+ 587us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:25.314860 (+ 1233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:25.314868 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:25.314882 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:53:25.314898 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:25.314974 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:53:25.314991 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:25.315326 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:25.315474 (+ 148us) server_negotiation.cc:299] Negotiation successful
0806 02:53:25.315765 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":691,"thread_start_us":414,"threads_started":1}
I20260806 02:53:25.320483 8716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a1b303caff746a1a0f3617c977a2156: Bootstrap starting.
I20260806 02:53:25.323822 8716 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a1b303caff746a1a0f3617c977a2156: Neither blocks nor log segments found. Creating new log.
I20260806 02:53:25.324908 8716 log.cc:824] T 00000000000000000000000000000000 P 3a1b303caff746a1a0f3617c977a2156: Log is configured to *not* fsync() on all Append() calls
I20260806 02:53:25.328114 8716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a1b303caff746a1a0f3617c977a2156: No bootstrap required, opened a new log
I20260806 02:53:25.335862 8716 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a1b303caff746a1a0f3617c977a2156 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a1b303caff746a1a0f3617c977a2156" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 37613 } }
I20260806 02:53:25.336303 8716 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a1b303caff746a1a0f3617c977a2156 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:53:25.336421 8716 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a1b303caff746a1a0f3617c977a2156 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a1b303caff746a1a0f3617c977a2156, State: Initialized, Role: FOLLOWER
I20260806 02:53:25.337046 8716 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a1b303caff746a1a0f3617c977a2156 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a1b303caff746a1a0f3617c977a2156" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 37613 } }
I20260806 02:53:25.337280 8716 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a1b303caff746a1a0f3617c977a2156 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:53:25.337425 8716 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a1b303caff746a1a0f3617c977a2156 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:53:25.337589 8716 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a1b303caff746a1a0f3617c977a2156 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:53:25.340271 8716 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a1b303caff746a1a0f3617c977a2156 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a1b303caff746a1a0f3617c977a2156" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 37613 } }
I20260806 02:53:25.340791 8716 leader_election.cc:302] T 00000000000000000000000000000000 P 3a1b303caff746a1a0f3617c977a2156 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a1b303caff746a1a0f3617c977a2156; no voters:
I20260806 02:53:25.341435 8716 leader_election.cc:288] T 00000000000000000000000000000000 P 3a1b303caff746a1a0f3617c977a2156 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:53:25.341536 8724 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a1b303caff746a1a0f3617c977a2156 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:53:25.342419 8724 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a1b303caff746a1a0f3617c977a2156 [term 1 LEADER]: Becoming Leader. State: Replica: 3a1b303caff746a1a0f3617c977a2156, State: Running, Role: LEADER
I20260806 02:53:25.343050 8724 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a1b303caff746a1a0f3617c977a2156 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a1b303caff746a1a0f3617c977a2156" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 37613 } }
I20260806 02:53:25.343693 8716 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a1b303caff746a1a0f3617c977a2156 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:53:25.345858 8725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a1b303caff746a1a0f3617c977a2156 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a1b303caff746a1a0f3617c977a2156" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a1b303caff746a1a0f3617c977a2156" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 37613 } } }
I20260806 02:53:25.346299 8725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a1b303caff746a1a0f3617c977a2156 [sys.catalog]: This master's current role is: LEADER
I20260806 02:53:25.346818 8726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a1b303caff746a1a0f3617c977a2156 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a1b303caff746a1a0f3617c977a2156. Latest consensus state: current_term: 1 leader_uuid: "3a1b303caff746a1a0f3617c977a2156" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a1b303caff746a1a0f3617c977a2156" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 37613 } } }
I20260806 02:53:25.347189 8726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a1b303caff746a1a0f3617c977a2156 [sys.catalog]: This master's current role is: LEADER
I20260806 02:53:25.349278 8732 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:53:25.355209 8732 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:53:25.367738 8732 catalog_manager.cc:1347] Generated new cluster ID: f03043ed4a6e41e493a4f4f67a5f4037
I20260806 02:53:25.367908 8732 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:53:25.389742 8732 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:53:25.390349 8732 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:53:25.401502 8732 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a1b303caff746a1a0f3617c977a2156: Generated new TSK 0
I20260806 02:53:25.402498 8732 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:53:25.420047 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.129:0
--local_ip_for_outbound_sockets=127.25.211.129
--webserver_interface=127.25.211.129
--webserver_port=0
--tserver_master_addrs=127.25.211.190:37613
--builtin_ntp_servers=127.25.211.148:42299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260806 02:53:25.779392 8750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:53:25.779707 8750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:53:25.779830 8750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:53:25.787563 8750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:53:25.787681 8750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:53:25.787833 8750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.129
I20260806 02:53:25.797380 8750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:42299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.211.129
--webserver_port=0
--tserver_master_addrs=127.25.211.190:37613
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.129
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:53:25.798870 8750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:53:25.800822 8750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:53:25.816329 8758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:25.816406 8760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:25.816362 8762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:53:25.817696 8750 server_base.cc:1034] running on GCE node
I20260806 02:53:25.818467 8750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:53:25.819655 8750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:53:25.820895 8750 hybrid_clock.cc:648] HybridClock initialized: now 1785984805820829 us; error 64 us; skew 500 ppm
I20260806 02:53:25.821246 8750 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:53:25.827301 8750 webserver.cc:460] Webserver started at http://127.25.211.129:39479/ using document root <none> and password file <none>
I20260806 02:53:25.828123 8750 fs_manager.cc:359] Metadata directory not provided
I20260806 02:53:25.828233 8750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:53:25.828562 8750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:53:25.834367 8750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-0/data/instance:
uuid: "12ef874386ba40caab770b5131dfd82e"
format_stamp: "Formatted at 2026-08-06 02:53:25 on dist-test-slave-zj8q"
server_key: "4423999213eef84a3146141955506d58"
server_key_iv: "493192b72f79208cb9d5ba71c52ac4a1"
server_key_version: "encryptionkey@0"
I20260806 02:53:25.835337 8750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-0/wal/instance:
uuid: "12ef874386ba40caab770b5131dfd82e"
format_stamp: "Formatted at 2026-08-06 02:53:25 on dist-test-slave-zj8q"
server_key: "4423999213eef84a3146141955506d58"
server_key_iv: "493192b72f79208cb9d5ba71c52ac4a1"
server_key_version: "encryptionkey@0"
I20260806 02:53:25.841678 8750 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260806 02:53:25.845896 8772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:25.847457 8750 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:53:25.847627 8750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-0/wal
uuid: "12ef874386ba40caab770b5131dfd82e"
format_stamp: "Formatted at 2026-08-06 02:53:25 on dist-test-slave-zj8q"
server_key: "4423999213eef84a3146141955506d58"
server_key_iv: "493192b72f79208cb9d5ba71c52ac4a1"
server_key_version: "encryptionkey@0"
I20260806 02:53:25.847865 8750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:53:25.854122 8750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:53:25.855113 8750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:53:25.855396 8750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:53:25.856442 8750 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:53:25.858013 8750 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:53:25.858114 8750 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:25.858253 8750 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:53:25.858341 8750 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:25.891379 8750 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.129:43275
I20260806 02:53:25.891451 8945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.129:43275 every 8 connection(s)
I20260806 02:53:25.892807 8750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
I20260806 02:53:25.894410 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 8750
I20260806 02:53:25.894634 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-0/wal/instance
I20260806 02:53:25.895269 26446 external_mini_cluster.cc:1428] Setting key 6e09b3b839c4d2601b6c3e337f7a4772
I20260806 02:53:25.898756 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.130:0
--local_ip_for_outbound_sockets=127.25.211.130
--webserver_interface=127.25.211.130
--webserver_port=0
--tserver_master_addrs=127.25.211.190:37613
--builtin_ntp_servers=127.25.211.148:42299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260806 02:53:25.903105 8949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:25.895681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:34861 (local address 127.25.211.190:37613)
0806 02:53:25.896189 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:25.896196 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:25.897832 (+ 1636us) server_negotiation.cc:407] Connection header received
0806 02:53:25.900282 (+ 2450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:25.900287 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:25.900359 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:25.900494 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:25.902245 (+ 1751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:25.902251 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:25.902257 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:53:25.902279 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:25.902329 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:53:25.902344 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:25.902460 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:25.902748 (+ 288us) server_negotiation.cc:299] Negotiation successful
0806 02:53:25.902809 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":253,"threads_started":1}
I20260806 02:53:25.904554 8953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:25.896300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:37613 (local address 127.25.211.129:34861)
0806 02:53:25.897604 (+ 1304us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:25.897713 (+ 109us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:25.899918 (+ 2205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:25.900791 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:25.900805 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:25.900940 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:25.901702 (+ 762us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:53:25.901705 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:25.901763 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:53:25.901900 (+ 137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:53:25.902462 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:53:25.902474 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:53:25.902483 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 02:53:25.902814 (+ 331us) client_negotiation.cc:240] Negotiation successful
0806 02:53:25.903222 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":427,"threads_started":1}
I20260806 02:53:25.906466 8947 heartbeater.cc:347] Connected to a master server at 127.25.211.190:37613
I20260806 02:53:25.906818 8947 heartbeater.cc:464] Registering TS with master...
I20260806 02:53:25.907514 8947 heartbeater.cc:511] Master 127.25.211.190:37613 requested a full tablet report, sending...
I20260806 02:53:25.909993 8656 ts_manager.cc:194] Registered new tserver with Master: 12ef874386ba40caab770b5131dfd82e (127.25.211.129:43275)
W20260806 02:53:26.260396 8954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:53:26.260708 8954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:53:26.260824 8954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:53:26.268457 8954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:53:26.268602 8954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:53:26.268729 8954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.130
I20260806 02:53:26.278571 8954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:42299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.211.130
--webserver_port=0
--tserver_master_addrs=127.25.211.190:37613
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.130
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:53:26.279893 8954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:53:26.281833 8954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:53:26.295874 8960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:26.295899 8965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:26.295918 8961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:53:26.296767 8954 server_base.cc:1034] running on GCE node
I20260806 02:53:26.297425 8954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:53:26.298434 8954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:53:26.299690 8954 hybrid_clock.cc:648] HybridClock initialized: now 1785984806299643 us; error 86 us; skew 500 ppm
I20260806 02:53:26.300087 8954 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:53:26.302940 8954 webserver.cc:460] Webserver started at http://127.25.211.130:42955/ using document root <none> and password file <none>
I20260806 02:53:26.303862 8954 fs_manager.cc:359] Metadata directory not provided
I20260806 02:53:26.304001 8954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:53:26.304399 8954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:53:26.308658 8954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-1/data/instance:
uuid: "ca593ca29b864fd6be86b7d0a7e00824"
format_stamp: "Formatted at 2026-08-06 02:53:26 on dist-test-slave-zj8q"
server_key: "58e44822ee49f7bc30550f1ea3ee1310"
server_key_iv: "64de626abbdc846c50a70ae1e16681f5"
server_key_version: "encryptionkey@0"
I20260806 02:53:26.309737 8954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-1/wal/instance:
uuid: "ca593ca29b864fd6be86b7d0a7e00824"
format_stamp: "Formatted at 2026-08-06 02:53:26 on dist-test-slave-zj8q"
server_key: "58e44822ee49f7bc30550f1ea3ee1310"
server_key_iv: "64de626abbdc846c50a70ae1e16681f5"
server_key_version: "encryptionkey@0"
I20260806 02:53:26.317083 8954 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260806 02:53:26.321787 8976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:26.323398 8954 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:53:26.323565 8954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-1/wal
uuid: "ca593ca29b864fd6be86b7d0a7e00824"
format_stamp: "Formatted at 2026-08-06 02:53:26 on dist-test-slave-zj8q"
server_key: "58e44822ee49f7bc30550f1ea3ee1310"
server_key_iv: "64de626abbdc846c50a70ae1e16681f5"
server_key_version: "encryptionkey@0"
I20260806 02:53:26.323789 8954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:53:26.329955 8954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:53:26.330881 8954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:53:26.331128 8954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:53:26.332026 8954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:53:26.333464 8954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:53:26.333554 8954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:26.333642 8954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:53:26.333695 8954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:26.366096 8954 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.130:46541
I20260806 02:53:26.366178 9147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.130:46541 every 8 connection(s)
I20260806 02:53:26.367525 8954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
I20260806 02:53:26.371927 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 8954
I20260806 02:53:26.372121 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-1/wal/instance
I20260806 02:53:26.372560 26446 external_mini_cluster.cc:1428] Setting key 72ce6208c463dd961a7f253489c4393a
I20260806 02:53:26.375662 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.131:0
--local_ip_for_outbound_sockets=127.25.211.131
--webserver_interface=127.25.211.131
--webserver_port=0
--tserver_master_addrs=127.25.211.190:37613
--builtin_ntp_servers=127.25.211.148:42299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260806 02:53:26.376317 8949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:26.370269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:56655 (local address 127.25.211.190:37613)
0806 02:53:26.370462 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:26.370468 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:26.371928 (+ 1460us) server_negotiation.cc:407] Connection header received
0806 02:53:26.373837 (+ 1909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:26.373842 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:26.373932 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:26.374033 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:26.375533 (+ 1500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:26.375537 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:26.375541 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:53:26.375558 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:26.375601 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:53:26.375620 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:26.375742 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:26.376053 (+ 311us) server_negotiation.cc:299] Negotiation successful
0806 02:53:26.376116 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 02:53:26.377653 9151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:26.370783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:37613 (local address 127.25.211.130:56655)
0806 02:53:26.371755 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:26.371805 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:26.373533 (+ 1728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:26.374324 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:26.374332 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:26.374429 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:26.375135 (+ 706us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:53:26.375138 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:26.375196 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:53:26.375330 (+ 134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:53:26.375735 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:53:26.375747 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:53:26.375756 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 02:53:26.376096 (+ 340us) client_negotiation.cc:240] Negotiation successful
0806 02:53:26.376388 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":292,"threads_started":1}
I20260806 02:53:26.379113 9148 heartbeater.cc:347] Connected to a master server at 127.25.211.190:37613
I20260806 02:53:26.379421 9148 heartbeater.cc:464] Registering TS with master...
I20260806 02:53:26.380034 9148 heartbeater.cc:511] Master 127.25.211.190:37613 requested a full tablet report, sending...
I20260806 02:53:26.381459 8656 ts_manager.cc:194] Registered new tserver with Master: ca593ca29b864fd6be86b7d0a7e00824 (127.25.211.130:46541)
W20260806 02:53:26.727583 9153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:53:26.727900 9153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:53:26.727999 9153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:53:26.735544 9153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:53:26.735666 9153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:53:26.735786 9153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.131
I20260806 02:53:26.745359 9153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:42299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.211.131
--webserver_port=0
--tserver_master_addrs=127.25.211.190:37613
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.131
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:53:26.746722 9153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:53:26.748585 9153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:53:26.762573 9163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:26.762513 9166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:26.762490 9160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:53:26.768340 9153 server_base.cc:1034] running on GCE node
I20260806 02:53:26.769083 9153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:53:26.770183 9153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:53:26.771428 9153 hybrid_clock.cc:648] HybridClock initialized: now 1785984806771371 us; error 48 us; skew 500 ppm
I20260806 02:53:26.771772 9153 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:53:26.774288 9153 webserver.cc:460] Webserver started at http://127.25.211.131:40093/ using document root <none> and password file <none>
I20260806 02:53:26.775079 9153 fs_manager.cc:359] Metadata directory not provided
I20260806 02:53:26.775202 9153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:53:26.775553 9153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:53:26.778837 9153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-2/data/instance:
uuid: "6c861485bb03472eac9222ca323bf797"
format_stamp: "Formatted at 2026-08-06 02:53:26 on dist-test-slave-zj8q"
server_key: "dc41684f500f10c2afe33094f0f8a0d6"
server_key_iv: "993051d27c4193fea1d41a6544083dcd"
server_key_version: "encryptionkey@0"
I20260806 02:53:26.779721 9153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-2/wal/instance:
uuid: "6c861485bb03472eac9222ca323bf797"
format_stamp: "Formatted at 2026-08-06 02:53:26 on dist-test-slave-zj8q"
server_key: "dc41684f500f10c2afe33094f0f8a0d6"
server_key_iv: "993051d27c4193fea1d41a6544083dcd"
server_key_version: "encryptionkey@0"
I20260806 02:53:26.785897 9153 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260806 02:53:26.789757 9174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:26.791453 9153 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260806 02:53:26.791694 9153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-2/wal
uuid: "6c861485bb03472eac9222ca323bf797"
format_stamp: "Formatted at 2026-08-06 02:53:26 on dist-test-slave-zj8q"
server_key: "dc41684f500f10c2afe33094f0f8a0d6"
server_key_iv: "993051d27c4193fea1d41a6544083dcd"
server_key_version: "encryptionkey@0"
I20260806 02:53:26.791946 9153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:53:26.798214 9153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:53:26.799170 9153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:53:26.799434 9153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:53:26.800386 9153 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:53:26.801929 9153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:53:26.802032 9153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:26.802145 9153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:53:26.802243 9153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260806 02:53:26.834797 9153 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.131:33865
I20260806 02:53:26.834864 9333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.131:33865 every 8 connection(s)
I20260806 02:53:26.836316 9153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
I20260806 02:53:26.845273 8949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:26.839049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:50527 (local address 127.25.211.190:37613)
0806 02:53:26.839347 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:26.839354 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:26.841029 (+ 1675us) server_negotiation.cc:407] Connection header received
0806 02:53:26.842858 (+ 1829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:26.842862 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:26.842937 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:26.843036 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:26.844486 (+ 1450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:26.844491 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:26.844494 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:53:26.844514 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:26.844551 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:53:26.844563 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:26.844671 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:26.844865 (+ 194us) server_negotiation.cc:299] Negotiation successful
0806 02:53:26.844940 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260806 02:53:26.846339 9337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:26.839762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:37613 (local address 127.25.211.131:50527)
0806 02:53:26.840840 (+ 1078us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:26.840887 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:26.842553 (+ 1666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:26.843271 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:26.843281 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:26.843386 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:26.844143 (+ 757us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:53:26.844146 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:26.844190 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:53:26.844302 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:53:26.844648 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:53:26.844655 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:53:26.844661 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 02:53:26.844914 (+ 253us) client_negotiation.cc:240] Negotiation successful
0806 02:53:26.845180 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":374,"threads_started":1}
I20260806 02:53:26.846693 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 9153
I20260806 02:53:26.846916 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1785984707115702-26446-0/minicluster-data/ts-2/wal/instance
I20260806 02:53:26.847447 26446 external_mini_cluster.cc:1428] Setting key f66b42657a253ae885c91abedad28afc
I20260806 02:53:26.847886 9335 heartbeater.cc:347] Connected to a master server at 127.25.211.190:37613
I20260806 02:53:26.848222 9335 heartbeater.cc:464] Registering TS with master...
I20260806 02:53:26.848937 9335 heartbeater.cc:511] Master 127.25.211.190:37613 requested a full tablet report, sending...
I20260806 02:53:26.850386 8656 ts_manager.cc:194] Registered new tserver with Master: 6c861485bb03472eac9222ca323bf797 (127.25.211.131:33865)
I20260806 02:53:26.863204 26446 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 02:53:26.869444 8949 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:53:26.867879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60004 (local address 127.25.211.190:37613)
0806 02:53:26.868133 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:26.868141 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:26.868515 (+ 374us) server_negotiation.cc:407] Connection header received
0806 02:53:26.868674 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:26.868678 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:26.868736 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:26.868854 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:26.869174 (+ 320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60004: BlockingRecv error: recv got EOF from 127.0.0.1:60004 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
I20260806 02:53:26.870342 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 8750
I20260806 02:53:26.874277 9351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:26.871456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46548 (local address 127.25.211.129:43275)
0806 02:53:26.871987 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:26.871995 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:26.872021 (+ 26us) server_negotiation.cc:407] Connection header received
0806 02:53:26.872081 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:26.872088 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:26.872333 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:26.872473 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:26.873458 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:26.873465 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:26.873476 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:53:26.873509 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:26.873552 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:53:26.873567 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:26.873690 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:26.873832 (+ 142us) server_negotiation.cc:299] Negotiation successful
0806 02:53:26.873937 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":249,"threads_started":1}
I20260806 02:53:26.875541 8940 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:53:26.958446 8947 heartbeater.cc:503] Master 127.25.211.190:37613 was elected leader, sending a full tablet report...
I20260806 02:53:26.960238 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 8750
I20260806 02:53:26.983400 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 8954
I20260806 02:53:26.987074 9354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:26.984320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44112 (local address 127.25.211.130:46541)
0806 02:53:26.984848 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:26.984855 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:26.984883 (+ 28us) server_negotiation.cc:407] Connection header received
0806 02:53:26.984948 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:26.984955 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:26.985181 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:26.985323 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:26.986316 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:26.986323 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:26.986336 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:53:26.986358 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:26.986404 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:53:26.986419 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:26.986539 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:26.986618 (+ 79us) server_negotiation.cc:299] Negotiation successful
0806 02:53:26.986720 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":255,"threads_started":1}
I20260806 02:53:26.988418 9138 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:53:27.071832 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 8954
I20260806 02:53:27.094944 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 9153
I20260806 02:53:27.099088 9356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:27.096138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54244 (local address 127.25.211.131:33865)
0806 02:53:27.096835 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:27.096843 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:27.096876 (+ 33us) server_negotiation.cc:407] Connection header received
0806 02:53:27.096970 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:27.096977 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:27.097222 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:27.097378 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:27.098432 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:27.098439 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:27.098452 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:53:27.098471 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:27.098513 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:53:27.098527 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:27.098663 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:27.098715 (+ 52us) server_negotiation.cc:299] Negotiation successful
0806 02:53:27.098803 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":356,"threads_started":1}
I20260806 02:53:27.100406 9329 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:53:27.182803 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 9153
I20260806 02:53:27.207526 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 8613
I20260806 02:53:27.208796 8710 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:53:27.313896 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 8613
2026-08-06T02:53:27Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2551 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:53:27 dist-test-slave-zj8q krb5kdc[9363](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:53:27 dist-test-slave-zj8q krb5kdc[9363](info): set up 2 sockets
Aug 06 02:53:27 dist-test-slave-zj8q krb5kdc[9363](info): commencing operation
krb5kdc: starting...
W20260806 02:53:28.547139 26446 mini_kdc.cc:121] Time spent starting KDC: real 1.186s user 0.001s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Aug 06 02:53:28 dist-test-slave-zj8q krb5kdc[9363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:53:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:53:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.211.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab.
Entry for principal kudu/127.25.211.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab.
I20260806 02:53:28.857954 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:39623
--webserver_interface=127.25.211.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:37465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.211.190:39623
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:53:29.213321 9381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:53:29.213649 9381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:53:29.213783 9381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:53:29.221292 9381 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:53:29.221431 9381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:53:29.221493 9381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:53:29.221585 9381 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:53:29.221652 9381 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:53:29.232120 9381 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:37465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.211.190:39623
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:39623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.211.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:53:29.233593 9381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:53:29.235013 9381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:53:29.247897 9389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:29.247927 9386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:29.247851 9387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:53:29.248842 9381 server_base.cc:1034] running on GCE node
I20260806 02:53:29.249750 9381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:53:29.251547 9381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:53:29.252796 9381 hybrid_clock.cc:648] HybridClock initialized: now 1785984809252725 us; error 56 us; skew 500 ppm
I20260806 02:53:29.253181 9381 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:53:29 dist-test-slave-zj8q krb5kdc[9363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984809, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:53:29.256531 9381 init.cc:377] Logged in from keytab as kudu/127.25.211.190@KRBTEST.COM (short username kudu)
I20260806 02:53:29.258381 9381 webserver.cc:460] Webserver started at http://127.25.211.190:42895/ using document root <none> and password file <none>
I20260806 02:53:29.259104 9381 fs_manager.cc:359] Metadata directory not provided
I20260806 02:53:29.259234 9381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:53:29.259552 9381 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:53:29.262835 9381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/master-0/data/instance:
uuid: "816c3420a6de47559f7635f307504c9b"
format_stamp: "Formatted at 2026-08-06 02:53:29 on dist-test-slave-zj8q"
server_key: "12b7ca7a3eb2fda6b6de2d599a7a785b"
server_key_iv: "1eb67b05d698f138c0946add98465ffc"
server_key_version: "encryptionkey@0"
I20260806 02:53:29.263652 9381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/master-0/wal/instance:
uuid: "816c3420a6de47559f7635f307504c9b"
format_stamp: "Formatted at 2026-08-06 02:53:29 on dist-test-slave-zj8q"
server_key: "12b7ca7a3eb2fda6b6de2d599a7a785b"
server_key_iv: "1eb67b05d698f138c0946add98465ffc"
server_key_version: "encryptionkey@0"
I20260806 02:53:29.269325 9381 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260806 02:53:29.273361 9401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:29.274865 9381 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:53:29.275055 9381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/master-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/master-0/wal
uuid: "816c3420a6de47559f7635f307504c9b"
format_stamp: "Formatted at 2026-08-06 02:53:29 on dist-test-slave-zj8q"
server_key: "12b7ca7a3eb2fda6b6de2d599a7a785b"
server_key_iv: "1eb67b05d698f138c0946add98465ffc"
server_key_version: "encryptionkey@0"
I20260806 02:53:29.275271 9381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:53:29.298364 9381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:53:29.302861 9381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:53:29.303158 9381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:53:29.321368 9381 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.190:39623
I20260806 02:53:29.321388 9487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.190:39623 every 8 connection(s)
I20260806 02:53:29.322774 9381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
I20260806 02:53:29.326593 9488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:53:29.328646 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 9381
I20260806 02:53:29.328904 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/master-0/wal/instance
I20260806 02:53:29.329730 26446 external_mini_cluster.cc:1428] Setting key 389de0501498d78c9cf40773b0505271
I20260806 02:53:29.339984 9488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 816c3420a6de47559f7635f307504c9b: Bootstrap starting.
I20260806 02:53:29.344017 9488 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 816c3420a6de47559f7635f307504c9b: Neither blocks nor log segments found. Creating new log.
Aug 06 02:53:29 dist-test-slave-zj8q krb5kdc[9363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:53:29.345286 9488 log.cc:824] T 00000000000000000000000000000000 P 816c3420a6de47559f7635f307504c9b: Log is configured to *not* fsync() on all Append() calls
I20260806 02:53:29.348670 9488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 816c3420a6de47559f7635f307504c9b: No bootstrap required, opened a new log
I20260806 02:53:29.353956 9494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:29.332084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52704 (local address 127.25.211.190:39623)
0806 02:53:29.332911 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:29.332924 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:29.332973 (+ 49us) server_negotiation.cc:407] Connection header received
0806 02:53:29.334921 (+ 1948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:29.334947 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:29.335379 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:29.335724 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:29.337563 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:29.339001 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:29.340789 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:29.341499 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:29.341714 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:29.346593 (+ 4879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:29.346606 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:29.346627 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:53:29.346675 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:29.349649 (+ 2974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:29.350512 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:29.350519 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:29.350526 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:29.350720 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:29.351115 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:29.351118 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:29.351120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:29.351728 (+ 608us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:29.351974 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:29.352411 (+ 437us) server_negotiation.cc:299] Negotiation successful
0806 02:53:29.352843 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":357,"threads_started":1}
I20260806 02:53:29.357520 9488 raft_consensus.cc:348] T 00000000000000000000000000000000 P 816c3420a6de47559f7635f307504c9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "816c3420a6de47559f7635f307504c9b" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 39623 } }
I20260806 02:53:29.357900 9488 raft_consensus.cc:374] T 00000000000000000000000000000000 P 816c3420a6de47559f7635f307504c9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:53:29.358002 9488 raft_consensus.cc:729] T 00000000000000000000000000000000 P 816c3420a6de47559f7635f307504c9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816c3420a6de47559f7635f307504c9b, State: Initialized, Role: FOLLOWER
I20260806 02:53:29.358625 9488 consensus_queue.cc:260] T 00000000000000000000000000000000 P 816c3420a6de47559f7635f307504c9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "816c3420a6de47559f7635f307504c9b" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 39623 } }
I20260806 02:53:29.358855 9488 raft_consensus.cc:388] T 00000000000000000000000000000000 P 816c3420a6de47559f7635f307504c9b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:53:29.358987 9488 raft_consensus.cc:482] T 00000000000000000000000000000000 P 816c3420a6de47559f7635f307504c9b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:53:29.359154 9488 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 816c3420a6de47559f7635f307504c9b [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:53:29.361763 9488 raft_consensus.cc:504] T 00000000000000000000000000000000 P 816c3420a6de47559f7635f307504c9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "816c3420a6de47559f7635f307504c9b" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 39623 } }
I20260806 02:53:29.362370 9488 leader_election.cc:302] T 00000000000000000000000000000000 P 816c3420a6de47559f7635f307504c9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 816c3420a6de47559f7635f307504c9b; no voters:
I20260806 02:53:29.363003 9488 leader_election.cc:288] T 00000000000000000000000000000000 P 816c3420a6de47559f7635f307504c9b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:53:29.363147 9497 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 816c3420a6de47559f7635f307504c9b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:53:29.364108 9497 raft_consensus.cc:686] T 00000000000000000000000000000000 P 816c3420a6de47559f7635f307504c9b [term 1 LEADER]: Becoming Leader. State: Replica: 816c3420a6de47559f7635f307504c9b, State: Running, Role: LEADER
I20260806 02:53:29.364797 9497 consensus_queue.cc:237] T 00000000000000000000000000000000 P 816c3420a6de47559f7635f307504c9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "816c3420a6de47559f7635f307504c9b" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 39623 } }
I20260806 02:53:29.365939 9488 sys_catalog.cc:564] T 00000000000000000000000000000000 P 816c3420a6de47559f7635f307504c9b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:53:29.367664 9499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 816c3420a6de47559f7635f307504c9b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "816c3420a6de47559f7635f307504c9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "816c3420a6de47559f7635f307504c9b" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 39623 } } }
I20260806 02:53:29.367977 9499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 816c3420a6de47559f7635f307504c9b [sys.catalog]: This master's current role is: LEADER
I20260806 02:53:29.369414 9498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 816c3420a6de47559f7635f307504c9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 816c3420a6de47559f7635f307504c9b. Latest consensus state: current_term: 1 leader_uuid: "816c3420a6de47559f7635f307504c9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "816c3420a6de47559f7635f307504c9b" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 39623 } } }
I20260806 02:53:29.369704 9498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 816c3420a6de47559f7635f307504c9b [sys.catalog]: This master's current role is: LEADER
I20260806 02:53:29.370090 9506 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:53:29.375047 9506 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:53:29.383841 9506 catalog_manager.cc:1347] Generated new cluster ID: 66df08ab7bba42b4bfeb201c5393e127
I20260806 02:53:29.383962 9506 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:53:29.397228 9506 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:53:29.398716 9506 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:53:29.413627 9506 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 816c3420a6de47559f7635f307504c9b: Generated new TSK 0
I20260806 02:53:29.414630 9506 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.211.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab.
Entry for principal kudu/127.25.211.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab.
I20260806 02:53:29.490926 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.129:0
--local_ip_for_outbound_sockets=127.25.211.129
--webserver_interface=127.25.211.129
--webserver_port=0
--tserver_master_addrs=127.25.211.190:39623
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab
--principal=kudu/127.25.211.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:37465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:53:29.844615 9531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:53:29.844935 9531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:53:29.845062 9531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:53:29.853122 9531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:53:29.853263 9531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:53:29.853415 9531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.129
I20260806 02:53:29.863909 9531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:37465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab
--principal=kudu/127.25.211.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.211.129
--webserver_port=0
--tserver_master_addrs=127.25.211.190:39623
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.129
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:53:29.865379 9531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:53:29.867059 9531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:53:29.881721 9538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:29.881855 9539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:29.881774 9541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:53:29.882987 9531 server_base.cc:1034] running on GCE node
I20260806 02:53:29.883672 9531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:53:29.884748 9531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:53:29.885998 9531 hybrid_clock.cc:648] HybridClock initialized: now 1785984809885925 us; error 62 us; skew 500 ppm
I20260806 02:53:29.886397 9531 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:53:29 dist-test-slave-zj8q krb5kdc[9363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984809, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:53:29.889902 9531 init.cc:377] Logged in from keytab as kudu/127.25.211.129@KRBTEST.COM (short username kudu)
I20260806 02:53:29.891645 9531 webserver.cc:460] Webserver started at http://127.25.211.129:38637/ using document root <none> and password file <none>
I20260806 02:53:29.892426 9531 fs_manager.cc:359] Metadata directory not provided
I20260806 02:53:29.892562 9531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:53:29.892868 9531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:53:29.896008 9531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/ts-0/data/instance:
uuid: "ad662f26f31b42e78def636b57069da3"
format_stamp: "Formatted at 2026-08-06 02:53:29 on dist-test-slave-zj8q"
server_key: "ab6eb8c87e4c74517300aa6628b3a9c5"
server_key_iv: "95f15dc5f20f1b449c050cfd9fb68f34"
server_key_version: "encryptionkey@0"
I20260806 02:53:29.896849 9531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/ts-0/wal/instance:
uuid: "ad662f26f31b42e78def636b57069da3"
format_stamp: "Formatted at 2026-08-06 02:53:29 on dist-test-slave-zj8q"
server_key: "ab6eb8c87e4c74517300aa6628b3a9c5"
server_key_iv: "95f15dc5f20f1b449c050cfd9fb68f34"
server_key_version: "encryptionkey@0"
I20260806 02:53:29.902443 9531 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260806 02:53:29.906443 9550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:29.908018 9531 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:53:29.908212 9531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/ts-0/wal
uuid: "ad662f26f31b42e78def636b57069da3"
format_stamp: "Formatted at 2026-08-06 02:53:29 on dist-test-slave-zj8q"
server_key: "ab6eb8c87e4c74517300aa6628b3a9c5"
server_key_iv: "95f15dc5f20f1b449c050cfd9fb68f34"
server_key_version: "encryptionkey@0"
I20260806 02:53:29.908418 9531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:53:29.920979 9531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:53:29.925806 9531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:53:29.926137 9531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:53:29.927277 9531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:53:29.928808 9531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:53:29.928891 9531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:29.929016 9531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:53:29.929071 9531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:29.963989 9531 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.129:43161
I20260806 02:53:29.964067 9743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.129:43161 every 8 connection(s)
I20260806 02:53:29.965432 9531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
I20260806 02:53:29.974476 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 9531
I20260806 02:53:29.974705 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/ts-0/wal/instance
I20260806 02:53:29.975373 26446 external_mini_cluster.cc:1428] Setting key 814492e254665e7b592a804c029983ef
Aug 06 02:53:29 dist-test-slave-zj8q krb5kdc[9363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984809, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.129@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:53:29.987006 9746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:29.968467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:55597 (local address 127.25.211.190:39623)
0806 02:53:29.968938 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:29.968942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:29.970032 (+ 1090us) server_negotiation.cc:407] Connection header received
0806 02:53:29.972176 (+ 2144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:29.972182 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:29.972299 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:29.972409 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:29.974133 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:29.975000 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:29.976828 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:29.977276 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:29.977393 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:29.981286 (+ 3893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:29.981292 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:29.981295 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:53:29.981330 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:29.983808 (+ 2478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:29.984587 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:29.984592 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:29.984594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:29.984725 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:29.985167 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:29.985171 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:29.985173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:29.985449 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:29.985599 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:29.986386 (+ 787us) server_negotiation.cc:299] Negotiation successful
0806 02:53:29.986629 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":244,"threads_started":1}
I20260806 02:53:29.988166 9747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:29.969076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:39623 (local address 127.25.211.129:55597)
0806 02:53:29.969845 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:29.969895 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:29.971828 (+ 1933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:29.972707 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:29.972716 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:29.973304 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:29.973935 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:29.973955 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:29.975172 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:29.975176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:29.976661 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:29.976682 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:29.977515 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:29.977523 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:29.977766 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:29.978482 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:29.978507 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:29.981027 (+ 2520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:53:29.983978 (+ 2951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:29.983988 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:29.984007 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:29.984463 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:29.984833 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:29.984838 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:29.984842 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:29.985046 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:29.985590 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:53:29.985598 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:53:29.985979 (+ 381us) client_negotiation.cc:769] Sending connection context
0806 02:53:29.986274 (+ 295us) client_negotiation.cc:240] Negotiation successful
0806 02:53:29.986866 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":298,"threads_started":1}
I20260806 02:53:29.989835 9744 heartbeater.cc:347] Connected to a master server at 127.25.211.190:39623
I20260806 02:53:29.990288 9744 heartbeater.cc:464] Registering TS with master...
I20260806 02:53:29.991202 9744 heartbeater.cc:511] Master 127.25.211.190:39623 requested a full tablet report, sending...
I20260806 02:53:29.993566 9424 ts_manager.cc:194] Registered new tserver with Master: ad662f26f31b42e78def636b57069da3 (127.25.211.129:43161)
I20260806 02:53:29.995168 9424 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.211.129@KRBTEST.COM'} at 127.25.211.129:55597
WARNING: no policy specified for kudu/127.25.211.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab.
Entry for principal kudu/127.25.211.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab.
I20260806 02:53:30.042215 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.130:0
--local_ip_for_outbound_sockets=127.25.211.130
--webserver_interface=127.25.211.130
--webserver_port=0
--tserver_master_addrs=127.25.211.190:39623
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab
--principal=kudu/127.25.211.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:37465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:53:30.401467 9750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:53:30.401814 9750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:53:30.401952 9750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:53:30.409780 9750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:53:30.409952 9750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:53:30.410115 9750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.130
I20260806 02:53:30.421638 9750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:37465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab
--principal=kudu/127.25.211.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.211.130
--webserver_port=0
--tserver_master_addrs=127.25.211.190:39623
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.130
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:53:30.423199 9750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:53:30.424762 9750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:53:30.438719 9757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:30.438825 9760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:30.438789 9758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:53:30.439481 9750 server_base.cc:1034] running on GCE node
I20260806 02:53:30.440112 9750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:53:30.441120 9750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:53:30.442339 9750 hybrid_clock.cc:648] HybridClock initialized: now 1785984810442279 us; error 41 us; skew 500 ppm
I20260806 02:53:30.442735 9750 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:53:30 dist-test-slave-zj8q krb5kdc[9363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984810, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:53:30.446133 9750 init.cc:377] Logged in from keytab as kudu/127.25.211.130@KRBTEST.COM (short username kudu)
I20260806 02:53:30.447752 9750 webserver.cc:460] Webserver started at http://127.25.211.130:46753/ using document root <none> and password file <none>
I20260806 02:53:30.448478 9750 fs_manager.cc:359] Metadata directory not provided
I20260806 02:53:30.448592 9750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:53:30.448892 9750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:53:30.453012 9750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/ts-1/data/instance:
uuid: "320221b78c244f5aafed79f6d8899fa2"
format_stamp: "Formatted at 2026-08-06 02:53:30 on dist-test-slave-zj8q"
server_key: "c445aea757b2cc8c435f83c9640ce55b"
server_key_iv: "866c1ba16b67416ce12deddc32e5c50e"
server_key_version: "encryptionkey@0"
I20260806 02:53:30.454371 9750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/ts-1/wal/instance:
uuid: "320221b78c244f5aafed79f6d8899fa2"
format_stamp: "Formatted at 2026-08-06 02:53:30 on dist-test-slave-zj8q"
server_key: "c445aea757b2cc8c435f83c9640ce55b"
server_key_iv: "866c1ba16b67416ce12deddc32e5c50e"
server_key_version: "encryptionkey@0"
I20260806 02:53:30.462230 9750 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20260806 02:53:30.466315 9769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:30.468381 9750 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:53:30.468581 9750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/ts-1/wal
uuid: "320221b78c244f5aafed79f6d8899fa2"
format_stamp: "Formatted at 2026-08-06 02:53:30 on dist-test-slave-zj8q"
server_key: "c445aea757b2cc8c435f83c9640ce55b"
server_key_iv: "866c1ba16b67416ce12deddc32e5c50e"
server_key_version: "encryptionkey@0"
I20260806 02:53:30.468804 9750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:53:30.493352 9750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:53:30.497831 9750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:53:30.498127 9750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:53:30.499300 9750 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:53:30.500768 9750 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:53:30.500855 9750 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:30.500981 9750 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:53:30.501044 9750 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:30.532953 9750 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.130:45339
I20260806 02:53:30.532995 9937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.130:45339 every 8 connection(s)
I20260806 02:53:30.534333 9750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
I20260806 02:53:30.544205 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 9750
I20260806 02:53:30.544430 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/ts-1/wal/instance
I20260806 02:53:30.545058 26446 external_mini_cluster.cc:1428] Setting key ee6f848d7d98e6a66975a9e34e26cf71
Aug 06 02:53:30 dist-test-slave-zj8q krb5kdc[9363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984810, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.130@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:53:30.556668 9941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:30.537024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:49451 (local address 127.25.211.190:39623)
0806 02:53:30.537484 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:30.537489 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:30.538530 (+ 1041us) server_negotiation.cc:407] Connection header received
0806 02:53:30.540476 (+ 1946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:30.540482 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:30.540571 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:30.540670 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:30.542576 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:30.543343 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:30.545759 (+ 2416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:30.546415 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:30.546543 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:30.550549 (+ 4006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:30.550554 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:30.550558 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:53:30.550599 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:30.553233 (+ 2634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:30.554232 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:30.554236 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:30.554238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:30.554393 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:30.554861 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:30.554864 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:30.554866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:30.555145 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:30.555256 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:30.556185 (+ 929us) server_negotiation.cc:299] Negotiation successful
0806 02:53:30.556387 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":222,"threads_started":1}
I20260806 02:53:30.558007 9945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:30.537577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:39623 (local address 127.25.211.130:49451)
0806 02:53:30.538369 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:30.538411 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:30.540182 (+ 1771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:30.540944 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:30.540952 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:30.541554 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:30.542357 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:30.542393 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:30.543515 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:30.543519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:30.545556 (+ 2037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:30.545584 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:30.546673 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:30.546681 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:30.546909 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:30.547685 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:30.547710 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:30.550266 (+ 2556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:53:30.553421 (+ 3155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:30.553433 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:30.553452 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:30.554059 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:30.554497 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:30.554501 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:30.554504 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:30.554761 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:30.555262 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:53:30.555270 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:53:30.555759 (+ 489us) client_negotiation.cc:769] Sending connection context
0806 02:53:30.556070 (+ 311us) client_negotiation.cc:240] Negotiation successful
0806 02:53:30.556631 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":281,"threads_started":1}
I20260806 02:53:30.559638 9939 heartbeater.cc:347] Connected to a master server at 127.25.211.190:39623
I20260806 02:53:30.559991 9939 heartbeater.cc:464] Registering TS with master...
I20260806 02:53:30.560982 9939 heartbeater.cc:511] Master 127.25.211.190:39623 requested a full tablet report, sending...
I20260806 02:53:30.562672 9424 ts_manager.cc:194] Registered new tserver with Master: 320221b78c244f5aafed79f6d8899fa2 (127.25.211.130:45339)
I20260806 02:53:30.563530 9424 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.211.130@KRBTEST.COM'} at 127.25.211.130:49451
WARNING: no policy specified for kudu/127.25.211.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab.
Entry for principal kudu/127.25.211.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab.
I20260806 02:53:30.611614 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.131:0
--local_ip_for_outbound_sockets=127.25.211.131
--webserver_interface=127.25.211.131
--webserver_port=0
--tserver_master_addrs=127.25.211.190:39623
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab
--principal=kudu/127.25.211.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:37465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:53:30.958266 9948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:53:30.958606 9948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:53:30.958758 9948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:53:30.967156 9948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:53:30.967280 9948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:53:30.967439 9948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.131
I20260806 02:53:30.978024 9948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:37465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab
--principal=kudu/127.25.211.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.211.131
--webserver_port=0
--tserver_master_addrs=127.25.211.190:39623
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.131
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:53:30.979480 9948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:53:30.980849 9948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:53:30.994419 9957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:30.994349 9956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:53:30.994386 9960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:53:30.995322 9948 server_base.cc:1034] running on GCE node
I20260806 02:53:30.995960 9948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:53:30.996906 9948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:53:30.998131 9948 hybrid_clock.cc:648] HybridClock initialized: now 1785984810998082 us; error 55 us; skew 500 ppm
I20260806 02:53:30.998513 9948 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:53:30.999074 9744 heartbeater.cc:503] Master 127.25.211.190:39623 was elected leader, sending a full tablet report...
Aug 06 02:53:31 dist-test-slave-zj8q krb5kdc[9363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984811, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:53:31.007246 9948 init.cc:377] Logged in from keytab as kudu/127.25.211.131@KRBTEST.COM (short username kudu)
I20260806 02:53:31.008822 9948 webserver.cc:460] Webserver started at http://127.25.211.131:33283/ using document root <none> and password file <none>
I20260806 02:53:31.009632 9948 fs_manager.cc:359] Metadata directory not provided
I20260806 02:53:31.009766 9948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:53:31.010068 9948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:53:31.013345 9948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/ts-2/data/instance:
uuid: "c1d87ea0f3a948c89b7c7e13b0bd7196"
format_stamp: "Formatted at 2026-08-06 02:53:31 on dist-test-slave-zj8q"
server_key: "5569eed0bbfee9986c59079f80760774"
server_key_iv: "6470dfd5d1f763a4280f65576ed62f4c"
server_key_version: "encryptionkey@0"
I20260806 02:53:31.014186 9948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/ts-2/wal/instance:
uuid: "c1d87ea0f3a948c89b7c7e13b0bd7196"
format_stamp: "Formatted at 2026-08-06 02:53:31 on dist-test-slave-zj8q"
server_key: "5569eed0bbfee9986c59079f80760774"
server_key_iv: "6470dfd5d1f763a4280f65576ed62f4c"
server_key_version: "encryptionkey@0"
I20260806 02:53:31.019578 9948 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 02:53:31.023200 9970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:31.024634 9948 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 02:53:31.024808 9948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/ts-2/wal
uuid: "c1d87ea0f3a948c89b7c7e13b0bd7196"
format_stamp: "Formatted at 2026-08-06 02:53:31 on dist-test-slave-zj8q"
server_key: "5569eed0bbfee9986c59079f80760774"
server_key_iv: "6470dfd5d1f763a4280f65576ed62f4c"
server_key_version: "encryptionkey@0"
I20260806 02:53:31.025002 9948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:53:31.050422 9948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:53:31.054998 9948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:53:31.055320 9948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:53:31.056372 9948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:53:31.057791 9948 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:53:31.057873 9948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:31.058001 9948 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:53:31.058055 9948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:53:31.089650 9948 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.131:33917
I20260806 02:53:31.089689 10144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.131:33917 every 8 connection(s)
I20260806 02:53:31.091041 9948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
I20260806 02:53:31.091987 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 9948
I20260806 02:53:31.092212 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/ts-2/wal/instance
I20260806 02:53:31.092753 26446 external_mini_cluster.cc:1428] Setting key 7f43c4fa91d4c3b246732db5aa5c2d5e
Aug 06 02:53:31 dist-test-slave-zj8q krb5kdc[9363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984811, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.131@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:53:31.113080 10151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:31.093930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:51951 (local address 127.25.211.190:39623)
0806 02:53:31.094424 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:31.094429 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:31.095655 (+ 1226us) server_negotiation.cc:407] Connection header received
0806 02:53:31.097737 (+ 2082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:31.097742 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:31.097849 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:31.097943 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:31.099646 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:31.100467 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:31.102084 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:31.102538 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:31.102656 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:31.106428 (+ 3772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:31.106433 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:31.106437 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:53:31.106476 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:31.109541 (+ 3065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:31.110410 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:31.110414 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:31.110417 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:31.110546 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:31.111109 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:31.111113 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:31.111129 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:31.111420 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:31.111591 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:31.112421 (+ 830us) server_negotiation.cc:299] Negotiation successful
0806 02:53:31.112672 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":244,"threads_started":1}
I20260806 02:53:31.114091 10152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:31.094625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:39623 (local address 127.25.211.131:51951)
0806 02:53:31.095496 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:31.095544 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:31.097451 (+ 1907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:31.098252 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:31.098268 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:31.098831 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:53:31.099464 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:31.099484 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:31.100629 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:31.100633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:31.101921 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:31.101936 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:31.102788 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:31.102796 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:31.103045 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:31.103743 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:53:31.103768 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:53:31.106133 (+ 2365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:53:31.109706 (+ 3573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:31.109717 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:31.109734 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:31.110283 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:31.110753 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:53:31.110759 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:53:31.110763 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:53:31.110982 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:53:31.111591 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:53:31.111600 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:53:31.112003 (+ 403us) client_negotiation.cc:769] Sending connection context
0806 02:53:31.112286 (+ 283us) client_negotiation.cc:240] Negotiation successful
0806 02:53:31.112886 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":328,"threads_started":1}
I20260806 02:53:31.115466 10147 heartbeater.cc:347] Connected to a master server at 127.25.211.190:39623
I20260806 02:53:31.115823 10147 heartbeater.cc:464] Registering TS with master...
I20260806 02:53:31.116612 10147 heartbeater.cc:511] Master 127.25.211.190:39623 requested a full tablet report, sending...
I20260806 02:53:31.118191 9424 ts_manager.cc:194] Registered new tserver with Master: c1d87ea0f3a948c89b7c7e13b0bd7196 (127.25.211.131:33917)
I20260806 02:53:31.119017 9424 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.211.131@KRBTEST.COM'} at 127.25.211.131:51951
I20260806 02:53:31.121243 26446 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:53:31.138677 10151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:31.125950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52712 (local address 127.25.211.190:39623)
0806 02:53:31.126124 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:31.126130 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:31.126568 (+ 438us) server_negotiation.cc:407] Connection header received
0806 02:53:31.126781 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:31.126785 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:31.126879 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:31.126973 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:31.128444 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:31.129358 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:31.130755 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:31.131174 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:31.131282 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:31.133678 (+ 2396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:31.133683 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:31.133687 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:53:31.133725 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:31.136193 (+ 2468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:31.136865 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:31.136868 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:31.136871 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:31.136996 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:31.137324 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:31.137327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:31.137329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:31.137652 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:31.137884 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:31.138204 (+ 320us) server_negotiation.cc:299] Negotiation successful
0806 02:53:31.138415 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 02:53:31.142885 9424 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52712:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:53:31.149803 9424 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:53:31.174207 10167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:31.162943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.130:45339 (local address 127.0.0.1:36220)
0806 02:53:31.164065 (+ 1122us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:31.164120 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:31.164307 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:31.165427 (+ 1120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:31.165435 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:31.165478 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:53:31.165885 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:31.165901 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:31.167983 (+ 2082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:31.167989 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:31.170572 (+ 2583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:31.170595 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:31.173057 (+ 2462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:31.173063 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:31.173303 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:31.173361 (+ 58us) client_negotiation.cc:769] Sending connection context
0806 02:53:31.173504 (+ 143us) client_negotiation.cc:240] Negotiation successful
0806 02:53:31.173751 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":839,"thread_start_us":1233,"threads_started":1}
I20260806 02:53:31.175238 10168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:31.163143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36220 (local address 127.25.211.130:45339)
0806 02:53:31.164245 (+ 1102us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:31.164255 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:31.164286 (+ 31us) server_negotiation.cc:407] Connection header received
0806 02:53:31.164632 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:31.164641 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:31.164913 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:31.165107 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:53:31.166104 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:31.167802 (+ 1698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:31.170809 (+ 3007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:31.172471 (+ 1662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:31.172664 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:31.172876 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:31.174658 (+ 1782us) server_negotiation.cc:299] Negotiation successful
0806 02:53:31.174890 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":932,"thread_start_us":784,"threads_started":1}
I20260806 02:53:31.181669 9839 tablet_service.cc:1467] Processing CreateTablet for tablet 4b368c1bc5674b82813faca2fcffbc46 (DEFAULT_TABLE table=test-table [id=e6302c435cf94ef19040ceb0bc2a19f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:53:31.183564 9839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b368c1bc5674b82813faca2fcffbc46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:53:31.186415 10173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:31.164978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.131:33917 (local address 127.0.0.1:47308)
0806 02:53:31.177171 (+ 12193us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:31.177188 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:31.177312 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:31.178438 (+ 1126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:31.178443 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:31.178469 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:53:31.178901 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:31.178914 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:31.181236 (+ 2322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:31.181241 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:31.183331 (+ 2090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:31.183351 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:31.185551 (+ 2200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:31.185558 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:31.185761 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:31.185797 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 02:53:31.185886 (+ 89us) client_negotiation.cc:240] Negotiation successful
0806 02:53:31.186031 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12075,"thread_start_us":11922,"threads_started":1}
I20260806 02:53:31.187544 10178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:31.165609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47308 (local address 127.25.211.131:33917)
0806 02:53:31.176719 (+ 11110us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:31.176730 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:31.177451 (+ 721us) server_negotiation.cc:407] Connection header received
0806 02:53:31.177586 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:31.177595 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:31.177870 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:31.178079 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:53:31.179225 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:31.181060 (+ 1835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:31.183545 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:31.184985 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:31.185179 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:31.185358 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:31.187028 (+ 1670us) server_negotiation.cc:299] Negotiation successful
0806 02:53:31.187219 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10915,"thread_start_us":10748,"threads_started":1}
I20260806 02:53:31.188337 10167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:31.163748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:43161 (local address 127.0.0.1:33132)
0806 02:53:31.175699 (+ 11951us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:31.175723 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:31.175845 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:31.180691 (+ 4846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:31.180696 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:31.180723 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:53:31.180949 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:31.180963 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:31.183532 (+ 2569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:31.183539 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:31.185681 (+ 2142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:31.185701 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:31.187555 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:31.187561 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:31.187738 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:31.187783 (+ 45us) client_negotiation.cc:769] Sending connection context
0806 02:53:31.187870 (+ 87us) client_negotiation.cc:240] Negotiation successful
0806 02:53:31.188020 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11832,"mutex_wait_us":574,"thread_start_us":22802,"threads_started":1}
I20260806 02:53:31.188411 10171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:31.164260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33132 (local address 127.25.211.129:43161)
0806 02:53:31.179830 (+ 15570us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:31.179838 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:31.179871 (+ 33us) server_negotiation.cc:407] Connection header received
0806 02:53:31.179970 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:31.179978 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:31.180254 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:31.181172 (+ 918us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:53:31.181438 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:31.183303 (+ 1865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:31.185916 (+ 2613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:31.187386 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:31.187577 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:31.187759 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:31.187926 (+ 167us) server_negotiation.cc:299] Negotiation successful
0806 02:53:31.188080 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15361,"thread_start_us":16142,"threads_started":1}
I20260806 02:53:31.191962 9643 tablet_service.cc:1467] Processing CreateTablet for tablet 4b368c1bc5674b82813faca2fcffbc46 (DEFAULT_TABLE table=test-table [id=e6302c435cf94ef19040ceb0bc2a19f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:53:31.191967 10047 tablet_service.cc:1467] Processing CreateTablet for tablet 4b368c1bc5674b82813faca2fcffbc46 (DEFAULT_TABLE table=test-table [id=e6302c435cf94ef19040ceb0bc2a19f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:53:31.193500 9643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b368c1bc5674b82813faca2fcffbc46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:53:31.193500 10047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b368c1bc5674b82813faca2fcffbc46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:53:31.197992 10179 tablet_bootstrap.cc:492] T 4b368c1bc5674b82813faca2fcffbc46 P 320221b78c244f5aafed79f6d8899fa2: Bootstrap starting.
I20260806 02:53:31.201637 10179 tablet_bootstrap.cc:654] T 4b368c1bc5674b82813faca2fcffbc46 P 320221b78c244f5aafed79f6d8899fa2: Neither blocks nor log segments found. Creating new log.
I20260806 02:53:31.203081 10179 log.cc:824] T 4b368c1bc5674b82813faca2fcffbc46 P 320221b78c244f5aafed79f6d8899fa2: Log is configured to *not* fsync() on all Append() calls
I20260806 02:53:31.206074 10179 tablet_bootstrap.cc:492] T 4b368c1bc5674b82813faca2fcffbc46 P 320221b78c244f5aafed79f6d8899fa2: No bootstrap required, opened a new log
I20260806 02:53:31.206410 10179 ts_tablet_manager.cc:1397] T 4b368c1bc5674b82813faca2fcffbc46 P 320221b78c244f5aafed79f6d8899fa2: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260806 02:53:31.208971 10181 tablet_bootstrap.cc:492] T 4b368c1bc5674b82813faca2fcffbc46 P ad662f26f31b42e78def636b57069da3: Bootstrap starting.
I20260806 02:53:31.210069 10183 tablet_bootstrap.cc:492] T 4b368c1bc5674b82813faca2fcffbc46 P c1d87ea0f3a948c89b7c7e13b0bd7196: Bootstrap starting.
I20260806 02:53:31.212998 10181 tablet_bootstrap.cc:654] T 4b368c1bc5674b82813faca2fcffbc46 P ad662f26f31b42e78def636b57069da3: Neither blocks nor log segments found. Creating new log.
I20260806 02:53:31.213454 10183 tablet_bootstrap.cc:654] T 4b368c1bc5674b82813faca2fcffbc46 P c1d87ea0f3a948c89b7c7e13b0bd7196: Neither blocks nor log segments found. Creating new log.
I20260806 02:53:31.214838 10183 log.cc:824] T 4b368c1bc5674b82813faca2fcffbc46 P c1d87ea0f3a948c89b7c7e13b0bd7196: Log is configured to *not* fsync() on all Append() calls
I20260806 02:53:31.215128 10181 log.cc:824] T 4b368c1bc5674b82813faca2fcffbc46 P ad662f26f31b42e78def636b57069da3: Log is configured to *not* fsync() on all Append() calls
I20260806 02:53:31.216933 10179 raft_consensus.cc:348] T 4b368c1bc5674b82813faca2fcffbc46 P 320221b78c244f5aafed79f6d8899fa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320221b78c244f5aafed79f6d8899fa2" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 45339 } } peers { permanent_uuid: "c1d87ea0f3a948c89b7c7e13b0bd7196" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 33917 } } peers { permanent_uuid: "ad662f26f31b42e78def636b57069da3" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 43161 } }
I20260806 02:53:31.217419 10179 raft_consensus.cc:374] T 4b368c1bc5674b82813faca2fcffbc46 P 320221b78c244f5aafed79f6d8899fa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:53:31.217581 10179 raft_consensus.cc:729] T 4b368c1bc5674b82813faca2fcffbc46 P 320221b78c244f5aafed79f6d8899fa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 320221b78c244f5aafed79f6d8899fa2, State: Initialized, Role: FOLLOWER
I20260806 02:53:31.217923 10181 tablet_bootstrap.cc:492] T 4b368c1bc5674b82813faca2fcffbc46 P ad662f26f31b42e78def636b57069da3: No bootstrap required, opened a new log
I20260806 02:53:31.218129 10183 tablet_bootstrap.cc:492] T 4b368c1bc5674b82813faca2fcffbc46 P c1d87ea0f3a948c89b7c7e13b0bd7196: No bootstrap required, opened a new log
I20260806 02:53:31.218286 10181 ts_tablet_manager.cc:1397] T 4b368c1bc5674b82813faca2fcffbc46 P ad662f26f31b42e78def636b57069da3: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260806 02:53:31.218438 10183 ts_tablet_manager.cc:1397] T 4b368c1bc5674b82813faca2fcffbc46 P c1d87ea0f3a948c89b7c7e13b0bd7196: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.005s
I20260806 02:53:31.218298 10179 consensus_queue.cc:260] T 4b368c1bc5674b82813faca2fcffbc46 P 320221b78c244f5aafed79f6d8899fa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320221b78c244f5aafed79f6d8899fa2" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 45339 } } peers { permanent_uuid: "c1d87ea0f3a948c89b7c7e13b0bd7196" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 33917 } } peers { permanent_uuid: "ad662f26f31b42e78def636b57069da3" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 43161 } }
I20260806 02:53:31.219488 9939 heartbeater.cc:503] Master 127.25.211.190:39623 was elected leader, sending a full tablet report...
I20260806 02:53:31.220221 10179 ts_tablet_manager.cc:1428] T 4b368c1bc5674b82813faca2fcffbc46 P 320221b78c244f5aafed79f6d8899fa2: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260806 02:53:31.227118 10181 raft_consensus.cc:348] T 4b368c1bc5674b82813faca2fcffbc46 P ad662f26f31b42e78def636b57069da3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320221b78c244f5aafed79f6d8899fa2" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 45339 } } peers { permanent_uuid: "c1d87ea0f3a948c89b7c7e13b0bd7196" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 33917 } } peers { permanent_uuid: "ad662f26f31b42e78def636b57069da3" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 43161 } }
I20260806 02:53:31.227677 10181 raft_consensus.cc:374] T 4b368c1bc5674b82813faca2fcffbc46 P ad662f26f31b42e78def636b57069da3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:53:31.227838 10181 raft_consensus.cc:729] T 4b368c1bc5674b82813faca2fcffbc46 P ad662f26f31b42e78def636b57069da3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad662f26f31b42e78def636b57069da3, State: Initialized, Role: FOLLOWER
I20260806 02:53:31.228436 10181 consensus_queue.cc:260] T 4b368c1bc5674b82813faca2fcffbc46 P ad662f26f31b42e78def636b57069da3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320221b78c244f5aafed79f6d8899fa2" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 45339 } } peers { permanent_uuid: "c1d87ea0f3a948c89b7c7e13b0bd7196" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 33917 } } peers { permanent_uuid: "ad662f26f31b42e78def636b57069da3" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 43161 } }
I20260806 02:53:31.228451 10183 raft_consensus.cc:348] T 4b368c1bc5674b82813faca2fcffbc46 P c1d87ea0f3a948c89b7c7e13b0bd7196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320221b78c244f5aafed79f6d8899fa2" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 45339 } } peers { permanent_uuid: "c1d87ea0f3a948c89b7c7e13b0bd7196" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 33917 } } peers { permanent_uuid: "ad662f26f31b42e78def636b57069da3" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 43161 } }
I20260806 02:53:31.228844 10183 raft_consensus.cc:374] T 4b368c1bc5674b82813faca2fcffbc46 P c1d87ea0f3a948c89b7c7e13b0bd7196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:53:31.229009 10183 raft_consensus.cc:729] T 4b368c1bc5674b82813faca2fcffbc46 P c1d87ea0f3a948c89b7c7e13b0bd7196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1d87ea0f3a948c89b7c7e13b0bd7196, State: Initialized, Role: FOLLOWER
I20260806 02:53:31.229701 10183 consensus_queue.cc:260] T 4b368c1bc5674b82813faca2fcffbc46 P c1d87ea0f3a948c89b7c7e13b0bd7196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320221b78c244f5aafed79f6d8899fa2" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 45339 } } peers { permanent_uuid: "c1d87ea0f3a948c89b7c7e13b0bd7196" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 33917 } } peers { permanent_uuid: "ad662f26f31b42e78def636b57069da3" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 43161 } }
I20260806 02:53:31.230053 10181 ts_tablet_manager.cc:1428] T 4b368c1bc5674b82813faca2fcffbc46 P ad662f26f31b42e78def636b57069da3: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
I20260806 02:53:31.234839 10147 heartbeater.cc:503] Master 127.25.211.190:39623 was elected leader, sending a full tablet report...
I20260806 02:53:31.235404 10183 ts_tablet_manager.cc:1428] T 4b368c1bc5674b82813faca2fcffbc46 P c1d87ea0f3a948c89b7c7e13b0bd7196: Time spent starting tablet: real 0.017s user 0.008s sys 0.008s
W20260806 02:53:31.286509 9940 tablet.cc:2369] T 4b368c1bc5674b82813faca2fcffbc46 P 320221b78c244f5aafed79f6d8899fa2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:53:31.342742 10149 tablet.cc:2369] T 4b368c1bc5674b82813faca2fcffbc46 P c1d87ea0f3a948c89b7c7e13b0bd7196: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:53:31.408177 10186 raft_consensus.cc:482] T 4b368c1bc5674b82813faca2fcffbc46 P 320221b78c244f5aafed79f6d8899fa2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:53:31.408490 10186 raft_consensus.cc:504] T 4b368c1bc5674b82813faca2fcffbc46 P 320221b78c244f5aafed79f6d8899fa2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320221b78c244f5aafed79f6d8899fa2" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 45339 } } peers { permanent_uuid: "c1d87ea0f3a948c89b7c7e13b0bd7196" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 33917 } } peers { permanent_uuid: "ad662f26f31b42e78def636b57069da3" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 43161 } }
I20260806 02:53:31.410383 10186 leader_election.cc:288] T 4b368c1bc5674b82813faca2fcffbc46 P 320221b78c244f5aafed79f6d8899fa2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1d87ea0f3a948c89b7c7e13b0bd7196 (127.25.211.131:33917), ad662f26f31b42e78def636b57069da3 (127.25.211.129:43161)
I20260806 02:53:31.417497 10171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:31.410946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:48401 (local address 127.25.211.129:43161)
0806 02:53:31.411380 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:31.411386 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:31.411715 (+ 329us) server_negotiation.cc:407] Connection header received
0806 02:53:31.411995 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:31.412001 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:31.412089 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:31.412175 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:53:31.412932 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:31.413760 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:31.415744 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:31.416536 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:31.416690 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:31.416765 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:31.417116 (+ 351us) server_negotiation.cc:299] Negotiation successful
0806 02:53:31.417248 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284}
I20260806 02:53:31.417520 10178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:31.410948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:59041 (local address 127.25.211.131:33917)
0806 02:53:31.411188 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:31.411195 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:31.412010 (+ 815us) server_negotiation.cc:407] Connection header received
0806 02:53:31.412196 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:31.412201 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:31.412276 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:31.412392 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:53:31.413059 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:31.413867 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:31.415732 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:31.416511 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:31.416659 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:31.416738 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:31.417147 (+ 409us) server_negotiation.cc:299] Negotiation successful
0806 02:53:31.417269 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260806 02:53:31.417696 10191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:31.410796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.131:33917 (local address 127.25.211.130:59041)
0806 02:53:31.411901 (+ 1105us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:31.411920 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:31.412054 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:31.412408 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:31.412411 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:31.412435 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:53:31.412904 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:31.412917 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:31.414007 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:31.414011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:31.415574 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:31.415589 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:31.416747 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:31.416757 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:31.416983 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:31.417010 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 02:53:31.417098 (+ 88us) client_negotiation.cc:240] Negotiation successful
0806 02:53:31.417231 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":996,"mutex_wait_us":71,"thread_start_us":781,"threads_started":1}
I20260806 02:53:31.417809 10190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:31.410776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:43161 (local address 127.25.211.130:48401)
0806 02:53:31.411600 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0806 02:53:31.411617 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:53:31.411835 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:53:31.412197 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:53:31.412202 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:53:31.412228 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:53:31.412724 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:31.412742 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:31.413900 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:31.413905 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:31.415580 (+ 1675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:53:31.415594 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:53:31.416774 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:53:31.416780 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:53:31.416928 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:31.416983 (+ 55us) client_negotiation.cc:769] Sending connection context
0806 02:53:31.417056 (+ 73us) client_negotiation.cc:240] Negotiation successful
0806 02:53:31.417192 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"mutex_wait_us":64,"thread_start_us":467,"threads_started":1}
I20260806 02:53:31.418883 10072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b368c1bc5674b82813faca2fcffbc46" candidate_uuid: "320221b78c244f5aafed79f6d8899fa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1d87ea0f3a948c89b7c7e13b0bd7196" is_pre_election: true
I20260806 02:53:31.418883 9675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b368c1bc5674b82813faca2fcffbc46" candidate_uuid: "320221b78c244f5aafed79f6d8899fa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad662f26f31b42e78def636b57069da3" is_pre_election: true
I20260806 02:53:31.419407 10072 raft_consensus.cc:2454] T 4b368c1bc5674b82813faca2fcffbc46 P c1d87ea0f3a948c89b7c7e13b0bd7196 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 320221b78c244f5aafed79f6d8899fa2 in term 0.
I20260806 02:53:31.419409 9675 raft_consensus.cc:2454] T 4b368c1bc5674b82813faca2fcffbc46 P ad662f26f31b42e78def636b57069da3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 320221b78c244f5aafed79f6d8899fa2 in term 0.
I20260806 02:53:31.420073 9774 leader_election.cc:302] T 4b368c1bc5674b82813faca2fcffbc46 P 320221b78c244f5aafed79f6d8899fa2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 320221b78c244f5aafed79f6d8899fa2, c1d87ea0f3a948c89b7c7e13b0bd7196; no voters:
I20260806 02:53:31.420481 10186 raft_consensus.cc:2781] T 4b368c1bc5674b82813faca2fcffbc46 P 320221b78c244f5aafed79f6d8899fa2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:53:31.420614 10186 raft_consensus.cc:482] T 4b368c1bc5674b82813faca2fcffbc46 P 320221b78c244f5aafed79f6d8899fa2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:53:31.420712 10186 raft_consensus.cc:3037] T 4b368c1bc5674b82813faca2fcffbc46 P 320221b78c244f5aafed79f6d8899fa2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:53:31.423028 10186 raft_consensus.cc:504] T 4b368c1bc5674b82813faca2fcffbc46 P 320221b78c244f5aafed79f6d8899fa2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320221b78c244f5aafed79f6d8899fa2" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 45339 } } peers { permanent_uuid: "c1d87ea0f3a948c89b7c7e13b0bd7196" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 33917 } } peers { permanent_uuid: "ad662f26f31b42e78def636b57069da3" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 43161 } }
I20260806 02:53:31.423655 10186 leader_election.cc:288] T 4b368c1bc5674b82813faca2fcffbc46 P 320221b78c244f5aafed79f6d8899fa2 [CANDIDATE]: Term 1 election: Requested vote from peers c1d87ea0f3a948c89b7c7e13b0bd7196 (127.25.211.131:33917), ad662f26f31b42e78def636b57069da3 (127.25.211.129:43161)
I20260806 02:53:31.424192 9675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b368c1bc5674b82813faca2fcffbc46" candidate_uuid: "320221b78c244f5aafed79f6d8899fa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad662f26f31b42e78def636b57069da3"
I20260806 02:53:31.424197 10072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b368c1bc5674b82813faca2fcffbc46" candidate_uuid: "320221b78c244f5aafed79f6d8899fa2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1d87ea0f3a948c89b7c7e13b0bd7196"
I20260806 02:53:31.424392 10072 raft_consensus.cc:3037] T 4b368c1bc5674b82813faca2fcffbc46 P c1d87ea0f3a948c89b7c7e13b0bd7196 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:53:31.424382 9675 raft_consensus.cc:3037] T 4b368c1bc5674b82813faca2fcffbc46 P ad662f26f31b42e78def636b57069da3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:53:31.426769 10072 raft_consensus.cc:2454] T 4b368c1bc5674b82813faca2fcffbc46 P c1d87ea0f3a948c89b7c7e13b0bd7196 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 320221b78c244f5aafed79f6d8899fa2 in term 1.
I20260806 02:53:31.426841 9675 raft_consensus.cc:2454] T 4b368c1bc5674b82813faca2fcffbc46 P ad662f26f31b42e78def636b57069da3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 320221b78c244f5aafed79f6d8899fa2 in term 1.
I20260806 02:53:31.427187 9774 leader_election.cc:302] T 4b368c1bc5674b82813faca2fcffbc46 P 320221b78c244f5aafed79f6d8899fa2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 320221b78c244f5aafed79f6d8899fa2, c1d87ea0f3a948c89b7c7e13b0bd7196; no voters:
I20260806 02:53:31.427501 10186 raft_consensus.cc:2781] T 4b368c1bc5674b82813faca2fcffbc46 P 320221b78c244f5aafed79f6d8899fa2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:53:31.427945 10186 raft_consensus.cc:686] T 4b368c1bc5674b82813faca2fcffbc46 P 320221b78c244f5aafed79f6d8899fa2 [term 1 LEADER]: Becoming Leader. State: Replica: 320221b78c244f5aafed79f6d8899fa2, State: Running, Role: LEADER
I20260806 02:53:31.428521 10186 consensus_queue.cc:237] T 4b368c1bc5674b82813faca2fcffbc46 P 320221b78c244f5aafed79f6d8899fa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320221b78c244f5aafed79f6d8899fa2" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 45339 } } peers { permanent_uuid: "c1d87ea0f3a948c89b7c7e13b0bd7196" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 33917 } } peers { permanent_uuid: "ad662f26f31b42e78def636b57069da3" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 43161 } }
I20260806 02:53:31.433099 9424 catalog_manager.cc:5495] T 4b368c1bc5674b82813faca2fcffbc46 P 320221b78c244f5aafed79f6d8899fa2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 320221b78c244f5aafed79f6d8899fa2 (127.25.211.130). New cstate: current_term: 1 leader_uuid: "320221b78c244f5aafed79f6d8899fa2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "320221b78c244f5aafed79f6d8899fa2" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 45339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c1d87ea0f3a948c89b7c7e13b0bd7196" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 33917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad662f26f31b42e78def636b57069da3" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 43161 } health_report { overall_health: UNKNOWN } } }
W20260806 02:53:31.469317 9745 tablet.cc:2369] T 4b368c1bc5674b82813faca2fcffbc46 P ad662f26f31b42e78def636b57069da3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:53:31.472592 10168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:31.466649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36232 (local address 127.25.211.130:45339)
0806 02:53:31.466871 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:31.466878 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:31.466902 (+ 24us) server_negotiation.cc:407] Connection header received
0806 02:53:31.467138 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:31.467145 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:31.467257 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:31.467385 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:53:31.467729 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:31.468580 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:31.470239 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:31.470671 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:31.470771 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:31.471350 (+ 579us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:53:31.471903 (+ 553us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:53:31.472022 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:31.472161 (+ 139us) server_negotiation.cc:299] Negotiation successful
0806 02:53:31.472275 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 02:53:31.481849 9675 raft_consensus.cc:1261] T 4b368c1bc5674b82813faca2fcffbc46 P ad662f26f31b42e78def636b57069da3 [term 1 FOLLOWER]: Refusing update from remote peer 320221b78c244f5aafed79f6d8899fa2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:53:31.481849 10072 raft_consensus.cc:1261] T 4b368c1bc5674b82813faca2fcffbc46 P c1d87ea0f3a948c89b7c7e13b0bd7196 [term 1 FOLLOWER]: Refusing update from remote peer 320221b78c244f5aafed79f6d8899fa2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:53:31.483304 10186 consensus_queue.cc:1035] T 4b368c1bc5674b82813faca2fcffbc46 P 320221b78c244f5aafed79f6d8899fa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1d87ea0f3a948c89b7c7e13b0bd7196" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 33917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:53:31.483656 10192 consensus_queue.cc:1035] T 4b368c1bc5674b82813faca2fcffbc46 P 320221b78c244f5aafed79f6d8899fa2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad662f26f31b42e78def636b57069da3" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 43161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:53:31.513967 9423 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52712:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:53:31.514302 9423 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52712:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:53:31.518615 9423 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 816c3420a6de47559f7635f307504c9b: Sending DeleteTablet for 3 replicas of tablet 4b368c1bc5674b82813faca2fcffbc46
I20260806 02:53:31.520087 9643 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b368c1bc5674b82813faca2fcffbc46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:53:31 UTC) from {username='kudu', principal='kudu/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:33132
I20260806 02:53:31.520099 10047 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b368c1bc5674b82813faca2fcffbc46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:53:31 UTC) from {username='kudu', principal='kudu/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:47308
I20260806 02:53:31.520923 9839 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b368c1bc5674b82813faca2fcffbc46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:53:31 UTC) from {username='kudu', principal='kudu/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:36220
I20260806 02:53:31.521852 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 9531
I20260806 02:53:31.535861 10215 tablet_replica.cc:330] T 4b368c1bc5674b82813faca2fcffbc46 P ad662f26f31b42e78def636b57069da3: stopping tablet replica
I20260806 02:53:31.536644 10215 raft_consensus.cc:2229] T 4b368c1bc5674b82813faca2fcffbc46 P ad662f26f31b42e78def636b57069da3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:53:31.537467 10215 raft_consensus.cc:2258] T 4b368c1bc5674b82813faca2fcffbc46 P ad662f26f31b42e78def636b57069da3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:53:31.540179 10215 ts_tablet_manager.cc:1905] T 4b368c1bc5674b82813faca2fcffbc46 P ad662f26f31b42e78def636b57069da3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:53:31.540601 10216 tablet_replica.cc:330] T 4b368c1bc5674b82813faca2fcffbc46 P 320221b78c244f5aafed79f6d8899fa2: stopping tablet replica
I20260806 02:53:31.546674 10214 tablet_replica.cc:330] T 4b368c1bc5674b82813faca2fcffbc46 P c1d87ea0f3a948c89b7c7e13b0bd7196: stopping tablet replica
I20260806 02:53:31.546914 10216 raft_consensus.cc:2229] T 4b368c1bc5674b82813faca2fcffbc46 P 320221b78c244f5aafed79f6d8899fa2 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:53:31.547170 10214 raft_consensus.cc:2229] T 4b368c1bc5674b82813faca2fcffbc46 P c1d87ea0f3a948c89b7c7e13b0bd7196 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:53:31.547760 10214 raft_consensus.cc:2258] T 4b368c1bc5674b82813faca2fcffbc46 P c1d87ea0f3a948c89b7c7e13b0bd7196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:53:31.547739 10216 raft_consensus.cc:2258] T 4b368c1bc5674b82813faca2fcffbc46 P 320221b78c244f5aafed79f6d8899fa2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:53:31.549901 10215 ts_tablet_manager.cc:1918] T 4b368c1bc5674b82813faca2fcffbc46 P ad662f26f31b42e78def636b57069da3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:53:31.550129 10215 log.cc:1196] T 4b368c1bc5674b82813faca2fcffbc46 P ad662f26f31b42e78def636b57069da3: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/ts-0/wal/wals/4b368c1bc5674b82813faca2fcffbc46
I20260806 02:53:31.550761 10215 ts_tablet_manager.cc:1939] T 4b368c1bc5674b82813faca2fcffbc46 P ad662f26f31b42e78def636b57069da3: Deleting consensus metadata
I20260806 02:53:31.552423 10216 ts_tablet_manager.cc:1905] T 4b368c1bc5674b82813faca2fcffbc46 P 320221b78c244f5aafed79f6d8899fa2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:53:31.552724 9403 catalog_manager.cc:4841] TS ad662f26f31b42e78def636b57069da3 (127.25.211.129:43161): tablet 4b368c1bc5674b82813faca2fcffbc46 (table test-table [id=e6302c435cf94ef19040ceb0bc2a19f4]) successfully deleted
I20260806 02:53:31.552949 10214 ts_tablet_manager.cc:1905] T 4b368c1bc5674b82813faca2fcffbc46 P c1d87ea0f3a948c89b7c7e13b0bd7196: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 06 02:53:31 dist-test-slave-zj8q krb5kdc[9363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.129@KRBTEST.COM
I20260806 02:53:31.560909 10214 ts_tablet_manager.cc:1918] T 4b368c1bc5674b82813faca2fcffbc46 P c1d87ea0f3a948c89b7c7e13b0bd7196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:53:31.561329 10214 log.cc:1196] T 4b368c1bc5674b82813faca2fcffbc46 P c1d87ea0f3a948c89b7c7e13b0bd7196: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/ts-2/wal/wals/4b368c1bc5674b82813faca2fcffbc46
I20260806 02:53:31.561947 10214 ts_tablet_manager.cc:1939] T 4b368c1bc5674b82813faca2fcffbc46 P c1d87ea0f3a948c89b7c7e13b0bd7196: Deleting consensus metadata
I20260806 02:53:31.564064 9406 catalog_manager.cc:4841] TS c1d87ea0f3a948c89b7c7e13b0bd7196 (127.25.211.131:33917): tablet 4b368c1bc5674b82813faca2fcffbc46 (table test-table [id=e6302c435cf94ef19040ceb0bc2a19f4]) successfully deleted
I20260806 02:53:31.569154 10171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:31.523141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33140 (local address 127.25.211.129:43161)
0806 02:53:31.523334 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:31.523340 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:31.542950 (+ 19610us) server_negotiation.cc:407] Connection header received
0806 02:53:31.543477 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:31.543483 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:31.543566 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:31.543639 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:31.545492 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:31.546664 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:31.548737 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:31.549411 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:31.549516 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:31.560783 (+ 11267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:31.560809 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:31.560832 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:53:31.560887 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:31.564984 (+ 4097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:31.566052 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:31.566064 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:31.566067 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:31.566315 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:31.566840 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:31.566845 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:31.566849 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:31.567481 (+ 632us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:31.567788 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:31.568480 (+ 692us) server_negotiation.cc:299] Negotiation successful
0806 02:53:31.568753 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260806 02:53:31.569826 10216 ts_tablet_manager.cc:1918] T 4b368c1bc5674b82813faca2fcffbc46 P 320221b78c244f5aafed79f6d8899fa2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:53:31.570089 10216 log.cc:1196] T 4b368c1bc5674b82813faca2fcffbc46 P 320221b78c244f5aafed79f6d8899fa2: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1785984707115702-26446-0/minicluster-data/ts-1/wal/wals/4b368c1bc5674b82813faca2fcffbc46
I20260806 02:53:31.570111 9737 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:53:31.570849 10216 ts_tablet_manager.cc:1939] T 4b368c1bc5674b82813faca2fcffbc46 P 320221b78c244f5aafed79f6d8899fa2: Deleting consensus metadata
I20260806 02:53:31.574016 9406 catalog_manager.cc:4841] TS 320221b78c244f5aafed79f6d8899fa2 (127.25.211.130:45339): tablet 4b368c1bc5674b82813faca2fcffbc46 (table test-table [id=e6302c435cf94ef19040ceb0bc2a19f4]) successfully deleted
I20260806 02:53:31.721838 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 9531
I20260806 02:53:31.748800 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 9750
Aug 06 02:53:31 dist-test-slave-zj8q krb5kdc[9363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.130@KRBTEST.COM
I20260806 02:53:31.767023 10168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:31.749573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36246 (local address 127.25.211.130:45339)
0806 02:53:31.749747 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:31.749752 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:31.749813 (+ 61us) server_negotiation.cc:407] Connection header received
0806 02:53:31.750002 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:31.750006 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:31.750072 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:31.750178 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:31.751834 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:31.752713 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:31.754502 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:31.754964 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:31.755082 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:31.760271 (+ 5189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:31.760290 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:31.760309 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:53:31.760361 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:31.763654 (+ 3293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:31.764608 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:31.764617 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:31.764619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:31.764778 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:31.765193 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:31.765197 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:31.765199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:31.765627 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:31.765972 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:31.766442 (+ 470us) server_negotiation.cc:299] Negotiation successful
0806 02:53:31.766675 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 02:53:31.767933 9930 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:53:31.884052 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 9750
I20260806 02:53:31.906769 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 9948
Aug 06 02:53:31 dist-test-slave-zj8q krb5kdc[9363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.131@KRBTEST.COM
I20260806 02:53:31.922079 10178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:53:31.907580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47310 (local address 127.25.211.131:33917)
0806 02:53:31.907763 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0806 02:53:31.907768 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:53:31.907790 (+ 22us) server_negotiation.cc:407] Connection header received
0806 02:53:31.907951 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:53:31.907955 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:53:31.908016 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:53:31.908099 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:53:31.909337 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:31.910103 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:31.911407 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:53:31.911834 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:53:31.911943 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:53:31.916603 (+ 4660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:53:31.916622 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:53:31.916640 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:53:31.916688 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:53:31.919385 (+ 2697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:31.920074 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:31.920081 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:31.920084 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:31.920258 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:53:31.920564 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:53:31.920567 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:53:31.920569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:53:31.920949 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:53:31.921214 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0806 02:53:31.921517 (+ 303us) server_negotiation.cc:299] Negotiation successful
0806 02:53:31.921735 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 02:53:31.922899 10138 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:53:32.013710 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 9948
I20260806 02:53:32.047542 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 9381
I20260806 02:53:32.048939 9475 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:53:32.166023 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 9381
2026-08-06T02:53:32Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4856 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:53:32 dist-test-slave-zj8q krb5kdc[10232](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:53:32 dist-test-slave-zj8q krb5kdc[10232](info): set up 2 sockets
Aug 06 02:53:32 dist-test-slave-zj8q krb5kdc[10232](info): commencing operation
krb5kdc: starting...
W20260806 02:53:33.407290 26446 mini_kdc.cc:121] Time spent starting KDC: real 1.190s user 0.000s sys 0.046s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Aug 06 02:53:33 dist-test-slave-zj8q krb5kdc[10232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:53:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:53:33Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.25.211.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.25.211.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.25.211.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/krb5kdc/rangeradmin_127.25.211.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.25.211.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/krb5kdc/rangeradmin_127.25.211.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.25.211.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.25.211.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.25.211.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/krb5kdc/rangerlookup_127.25.211.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.25.211.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/krb5kdc/rangerlookup_127.25.211.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.25.211.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.211.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.211.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/krb5kdc/HTTP_127.25.211.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.25.211.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/krb5kdc/HTTP_127.25.211.148@KRBTEST.COM.keytab.
I20260806 02:53:33.840924 26446 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskFsNc4K/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/postgres -l logfile start
2026-08-06 02:53:36.477 UTC [10277] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-06 02:53:36.477 UTC [10277] LOG: listening on IPv4 address "127.25.211.148", port 33557
2026-08-06 02:53:36.481 UTC [10277] LOG: listening on Unix socket "/tmp/.s.PGSQL.33557"
2026-08-06 02:53:36.502 UTC [10281] LOG: database system was shut down at 2026-08-06 02:53:34 UTC
2026-08-06 02:53:36.506 UTC [10277] LOG: database system is ready to accept connections
I20260806 02:53:37.599027 26446 mini_postgres.cc:96] Postgres bound to 33557
2026-08-06 02:53:37.625 UTC [10296] FATAL: database "slave" does not exist
127.25.211.148:33557 - accepting connections
I20260806 02:53:37.626880 26446 mini_ranger.cc:161] Starting Ranger...
I20260806 02:53:37.662904 26446 mini_ranger.cc:84] Created miniranger Postgres user
I20260806 02:53:38.188715 26446 mini_ranger.cc:87] Created ranger Postgres database
I20260806 02:53:38.188864 26446 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ranger-admin
2026-08-06 02:53:39,278 [I] DB FLAVOR :POSTGRES
2026-08-06 02:53:39,278 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 02:53:39,279 [I] Checking connection..
2026-08-06 02:53:39,279 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFsNc4K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFsNc4K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.211.148:33557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-06 02:53:39,537 [I] Checking connection passed.
2026-08-06 02:53:39,537 [I] --------- Verifying version history table ---------
2026-08-06 02:53:39,537 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFsNc4K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFsNc4K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.211.148:33557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 02:53:39,812 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 02:53:39,812 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFsNc4K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFsNc4K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.211.148:33557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 02:53:40,085 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 02:53:40,085 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-06 02:53:40,085 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFsNc4K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFsNc4K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.211.148:33557/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskFsNc4K/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-06 02:53:40.355 UTC [10394] WARNING: there is no transaction in progress
2026-08-06 02:53:40,361 [I] create_dbversion_catalog.sql file imported successfully
2026-08-06 02:53:40,362 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFsNc4K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFsNc4K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.211.148:33557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 02:53:40,625 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-06 02:53:40,625 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 02:53:40,625 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFsNc4K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFsNc4K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.211.148:33557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-06 02:53:40,897 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFsNc4K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFsNc4K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.211.148:33557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-06 02:53:41,173 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFsNc4K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFsNc4K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.211.148:33557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-zj8q.c.gcp-upstream.internal','N') ;"
2026-08-06 02:53:41,429 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFsNc4K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFsNc4K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.211.148:33557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 02:53:41,692 [I] Table x_portal_user does not exist in database ranger
2026-08-06 02:53:41,692 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFsNc4K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFsNc4K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.211.148:33557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 02:53:41,961 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-06 02:53:41,961 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFsNc4K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFsNc4K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.211.148:33557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 02:53:42,207 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFsNc4K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFsNc4K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.211.148:33557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 02:53:42,475 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-06 02:53:42,476 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFsNc4K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFsNc4K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.211.148:33557/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskFsNc4K/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-06 02:53:43.465 UTC [10554] WARNING: there is no transaction in progress
2026-08-06 02:53:43.481 UTC [10554] WARNING: there is no transaction in progress
2026-08-06 02:53:43.498 UTC [10554] WARNING: there is no transaction in progress
2026-08-06 02:53:43.513 UTC [10554] WARNING: there is no transaction in progress
2026-08-06 02:53:43.532 UTC [10554] WARNING: there is no transaction in progress
2026-08-06 02:53:43.548 UTC [10554] WARNING: there is no transaction in progress
2026-08-06 02:53:43.610 UTC [10554] WARNING: there is no transaction in progress
2026-08-06 02:53:43.617 UTC [10554] WARNING: there is no transaction in progress
2026-08-06 02:53:43.633 UTC [10554] WARNING: there is no transaction in progress
2026-08-06 02:53:43.644 UTC [10554] WARNING: there is no transaction in progress
2026-08-06 02:53:43.656 UTC [10554] WARNING: there is no transaction in progress
2026-08-06 02:53:43.672 UTC [10554] WARNING: there is no transaction in progress
2026-08-06 02:53:43.683 UTC [10554] WARNING: there is no transaction in progress
2026-08-06 02:53:43.691 UTC [10554] WARNING: there is no transaction in progress
2026-08-06 02:53:43.699 UTC [10554] WARNING: there is no transaction in progress
2026-08-06 02:53:43.707 UTC [10554] WARNING: there is no transaction in progress
2026-08-06 02:53:44,399 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-06 02:53:44,399 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFsNc4K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFsNc4K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.211.148:33557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-06 02:53:44,654 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-06 02:53:44,654 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFsNc4K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFsNc4K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.211.148:33557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 02:53:44,910 [I] Table x_portal_user already exists in database 'ranger'
2026-08-06 02:53:44,910 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFsNc4K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFsNc4K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.211.148:33557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 02:53:45,170 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-06 02:53:45,171 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFsNc4K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFsNc4K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.211.148:33557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 02:53:45,439 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFsNc4K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFsNc4K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.211.148:33557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 02:53:45,689 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFsNc4K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFsNc4K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.211.148:33557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-zj8q.c.gcp-upstream.internal';"
2026-08-06 02:53:45,941 [I] CORE_DB_SCHEMA import status has been updated
2026-08-06 02:53:45,942 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFsNc4K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFsNc4K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.211.148:33557/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-06 02:53:46,212 [I] DB_PATCHES have already been applied
I20260806 02:53:46.218287 26446 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ranger-admin:/tmp/dist-test-taskFsNc4K/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskFsNc4K/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskFsNc4K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFsNc4K/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260806 02:53:46.218394 26446 mini_ranger.cc:193] Using host: 127.25.211.148
I20260806 02:53:46.241525 26446 mini_ranger.cc:227] Ranger admin URL: http://127.25.211.148:60979
Aug 06, 2026 2:53:46 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 2:53:46 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 2:53:46 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ranger-admin/ews/webapp
Aug 06, 2026 2:53:46 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 06, 2026 2:53:46 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 06, 2026 2:53:47 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ranger-admin/ews/webapp].
Aug 06, 2026 2:53:47 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260806 02:53:47.553753 26446 mini_ranger.cc:160] Time spent starting Ranger: real 9.927s user 0.003s sys 0.072s
Aug 06, 2026 2:54:07 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 06, 2026 2:54:07 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 06, 2026 2:54:07 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 06, 2026 2:54:07 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 06, 2026 2:54:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 06, 2026 2:54:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 06, 2026 2:54:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 06, 2026 2:54:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 06, 2026 2:54:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 06, 2026 2:54:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 06, 2026 2:54:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 06, 2026 2:54:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 06, 2026 2:54:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 06, 2026 2:54:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 06, 2026 2:54:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 06, 2026 2:54:07 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 06, 2026 2:54:07 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 06, 2026 2:54:08 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260806 02:54:08.749665 26446 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.25.211.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.25.211.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.25.211.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/krb5kdc/rangerkms_127.25.211.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.25.211.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/krb5kdc/rangerkms_127.25.211.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.25.211.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.25.211.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.211.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/krb5kdc/HTTP_127.25.211.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.25.211.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/krb5kdc/HTTP_127.25.211.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 06 02:54:08 dist-test-slave-zj8q krb5kdc[10232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984848, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260806 02:54:08.974428 26446 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260806 02:54:09.002847 26446 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260806 02:54:09.884841 26446 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260806 02:54:09.884997 26446 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ranger-kms
I20260806 02:54:09.885104 26446 mini_ranger_kms.cc:227] Using postgres at 127.25.211.148:33557
2026-08-06 02:54:10,718 [I] DB FLAVOR :POSTGRES
2026-08-06 02:54:10,719 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 02:54:10,719 [I] Checking connection
2026-08-06 02:54:10,719 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFsNc4K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.211.148:33557/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-06 02:54:10,983 [I] connection success
2026-08-06 02:54:10,983 [I] --------- Verifying Ranger DB tables ---------
2026-08-06 02:54:10,983 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-06 02:54:10,983 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFsNc4K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.211.148:33557/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-06 02:54:11,241 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-06 02:54:11,242 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 02:54:11,242 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-06 02:54:11,242 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFsNc4K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.211.148:33557/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-06 02:54:11,535 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260806 02:54:12.038838 26446 mini_ranger_kms.cc:310] Created kms service
I20260806 02:54:12.245213 26446 mini_ranger_kms.cc:326] Created kudu user
I20260806 02:54:12.290654 26446 mini_ranger_kms.cc:343] Created rangerkms user
Aug 06 02:54:12 dist-test-slave-zj8q krb5kdc[10232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984848, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.25.211.148@KRBTEST.COM
I20260806 02:54:12.656469 26446 mini_ranger_kms.cc:384] Added ranger policy
I20260806 02:54:12.657711 26446 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ranger-kms:/tmp/dist-test-taskFsNc4K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFsNc4K/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskFsNc4K/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260806 02:54:12.657773 26446 mini_ranger_kms.cc:240] Using host: 127.25.211.148
I20260806 02:54:12.690402 26446 mini_ranger_kms.cc:276] Ranger KMS PID: 10827
I20260806 02:54:12.690585 26446 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.25.211.148:54765
Aug 06, 2026 2:54:13 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 2:54:13 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 2:54:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 06, 2026 2:54:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 06, 2026 2:54:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ranger-kms/ews/webapp].
Aug 06, 2026 2:54:14 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 06 02:54:15 dist-test-slave-zj8q krb5kdc[10232](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785984855, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.25.211.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:54:15.298697 26446 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.324s user 0.136s sys 0.475s
I20260806 02:54:15.299090 26446 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260806 02:54:15.299311 26446 mini_ranger_kms.cc:401] 127.25.211.148:54765/kms/v1/keys
Aug 06 02:54:15 dist-test-slave-zj8q krb5kdc[10232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984855, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.25.211.148@KRBTEST.COM for HTTP/127.25.211.148@KRBTEST.COM
Aug 06 02:54:15 dist-test-slave-zj8q krb5kdc[10232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984855, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.25.211.148@KRBTEST.COM for HTTP/127.25.211.148@KRBTEST.COM
Aug 06 02:54:18 dist-test-slave-zj8q krb5kdc[10232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.25.211.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab.
Entry for principal kudu/127.25.211.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab.
I20260806 02:54:18.658564 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:33071
--webserver_interface=127.25.211.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:46835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.211.190:33071
--ranger_config_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.211.148:54765/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:54:19.146593 10888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:54:19.147075 10888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:54:19.147280 10888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:54:19.156734 10888 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:54:19.156883 10888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:54:19.156970 10888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:54:19.157047 10888 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:54:19.157171 10888 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:54:19.169992 10888 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:46835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/master-0/wal
--ranger_kms_url=127.25.211.148:54765/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.25.211.190:33071
--ranger_config_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:33071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.211.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:54:19.171837 10888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:54:19.173295 10888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:54:19.187024 10897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:19.187206 10896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:54:19.187460 10888 server_base.cc:1034] running on GCE node
W20260806 02:54:19.187050 10900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:54:19.188791 10888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:54:19.190477 10888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:54:19.191769 10888 hybrid_clock.cc:648] HybridClock initialized: now 1785984859191651 us; error 38 us; skew 500 ppm
I20260806 02:54:19.192284 10888 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:54:19 dist-test-slave-zj8q krb5kdc[10232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984859, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:54:19.200205 10888 init.cc:377] Logged in from keytab as kudu/127.25.211.190@KRBTEST.COM (short username kudu)
I20260806 02:54:19.201993 10888 webserver.cc:460] Webserver started at http://127.25.211.190:34789/ using document root <none> and password file <none>
I20260806 02:54:19.202790 10888 fs_manager.cc:359] Metadata directory not provided
I20260806 02:54:19.202929 10888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:54:19.203272 10888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 02:54:19 dist-test-slave-zj8q krb5kdc[10232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984859, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.190@KRBTEST.COM for HTTP/127.25.211.148@KRBTEST.COM
I20260806 02:54:19.262097 10888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/master-0/data/instance:
uuid: "c2f6e3bdb70e4dcebb69d5effeff7f83"
format_stamp: "Formatted at 2026-08-06 02:54:19 on dist-test-slave-zj8q"
server_key: "696130d37bcd5e163c5a07fde93fe30f"
server_key_iv: "b4209b008f34af1d4af628a9aa19de97"
server_key_version: "kuduclusterkey@0"
I20260806 02:54:19.263152 10888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/master-0/wal/instance:
uuid: "c2f6e3bdb70e4dcebb69d5effeff7f83"
format_stamp: "Formatted at 2026-08-06 02:54:19 on dist-test-slave-zj8q"
server_key: "696130d37bcd5e163c5a07fde93fe30f"
server_key_iv: "b4209b008f34af1d4af628a9aa19de97"
server_key_version: "kuduclusterkey@0"
I20260806 02:54:19.270000 10888 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260806 02:54:19.287544 10910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:19.289206 10888 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260806 02:54:19.289489 10888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/master-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/master-0/wal
uuid: "c2f6e3bdb70e4dcebb69d5effeff7f83"
format_stamp: "Formatted at 2026-08-06 02:54:19 on dist-test-slave-zj8q"
server_key: "696130d37bcd5e163c5a07fde93fe30f"
server_key_iv: "b4209b008f34af1d4af628a9aa19de97"
server_key_version: "kuduclusterkey@0"
I20260806 02:54:19.289763 10888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:54:19.311431 10888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:54:19.316504 10888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:54:19.316886 10888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:54:19.351769 10888 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.190:33071
I20260806 02:54:19.351851 10996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.190:33071 every 8 connection(s)
I20260806 02:54:19.353276 10888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
I20260806 02:54:19.358834 10999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:54:19.358820 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 10888
I20260806 02:54:19.359051 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/master-0/wal/instance
I20260806 02:54:19.359928 26446 external_mini_cluster.cc:1428] Setting key 434b1af951e7743c16702dd7c315c925
I20260806 02:54:19.377260 10999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2f6e3bdb70e4dcebb69d5effeff7f83: Bootstrap starting.
I20260806 02:54:19.382987 10999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2f6e3bdb70e4dcebb69d5effeff7f83: Neither blocks nor log segments found. Creating new log.
I20260806 02:54:19.384735 10999 log.cc:824] T 00000000000000000000000000000000 P c2f6e3bdb70e4dcebb69d5effeff7f83: Log is configured to *not* fsync() on all Append() calls
I20260806 02:54:19.392179 10999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2f6e3bdb70e4dcebb69d5effeff7f83: No bootstrap required, opened a new log
Aug 06 02:54:19 dist-test-slave-zj8q krb5kdc[10232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:54:19.402823 11002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:19.362258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42926 (local address 127.25.211.190:33071)
0806 02:54:19.377101 (+ 14843us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:19.377120 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:19.377228 (+ 108us) server_negotiation.cc:407] Connection header received
0806 02:54:19.381011 (+ 3783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:19.381047 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:19.381624 (+ 577us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:19.382137 (+ 513us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:54:19.383916 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:19.386441 (+ 2525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:19.388553 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:19.389351 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:19.389553 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:19.393781 (+ 4228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:54:19.393792 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:54:19.393875 (+ 83us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:54:19.393930 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:54:19.397877 (+ 3947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:19.398708 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:19.398716 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:19.398724 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:19.398886 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:19.399382 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:19.399388 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:19.399391 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:19.400162 (+ 771us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:54:19.400557 (+ 395us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:19.401052 (+ 495us) server_negotiation.cc:299] Negotiation successful
0806 02:54:19.401532 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14502,"thread_start_us":15113,"threads_started":1}
I20260806 02:54:19.407908 10999 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2f6e3bdb70e4dcebb69d5effeff7f83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2f6e3bdb70e4dcebb69d5effeff7f83" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 33071 } }
I20260806 02:54:19.408483 10999 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2f6e3bdb70e4dcebb69d5effeff7f83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:54:19.408604 10999 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2f6e3bdb70e4dcebb69d5effeff7f83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2f6e3bdb70e4dcebb69d5effeff7f83, State: Initialized, Role: FOLLOWER
I20260806 02:54:19.409475 10999 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2f6e3bdb70e4dcebb69d5effeff7f83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2f6e3bdb70e4dcebb69d5effeff7f83" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 33071 } }
I20260806 02:54:19.409812 10999 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2f6e3bdb70e4dcebb69d5effeff7f83 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:54:19.409936 10999 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2f6e3bdb70e4dcebb69d5effeff7f83 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:54:19.410079 10999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2f6e3bdb70e4dcebb69d5effeff7f83 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:54:19.412889 10999 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2f6e3bdb70e4dcebb69d5effeff7f83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2f6e3bdb70e4dcebb69d5effeff7f83" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 33071 } }
I20260806 02:54:19.413587 10999 leader_election.cc:302] T 00000000000000000000000000000000 P c2f6e3bdb70e4dcebb69d5effeff7f83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2f6e3bdb70e4dcebb69d5effeff7f83; no voters:
I20260806 02:54:19.414376 10999 leader_election.cc:288] T 00000000000000000000000000000000 P c2f6e3bdb70e4dcebb69d5effeff7f83 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:54:19.414565 11005 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2f6e3bdb70e4dcebb69d5effeff7f83 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:54:19.415488 11005 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2f6e3bdb70e4dcebb69d5effeff7f83 [term 1 LEADER]: Becoming Leader. State: Replica: c2f6e3bdb70e4dcebb69d5effeff7f83, State: Running, Role: LEADER
I20260806 02:54:19.416133 11005 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2f6e3bdb70e4dcebb69d5effeff7f83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2f6e3bdb70e4dcebb69d5effeff7f83" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 33071 } }
I20260806 02:54:19.416993 10999 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2f6e3bdb70e4dcebb69d5effeff7f83 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:54:19.420607 11006 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2f6e3bdb70e4dcebb69d5effeff7f83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2f6e3bdb70e4dcebb69d5effeff7f83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2f6e3bdb70e4dcebb69d5effeff7f83" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 33071 } } }
I20260806 02:54:19.421682 11006 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2f6e3bdb70e4dcebb69d5effeff7f83 [sys.catalog]: This master's current role is: LEADER
I20260806 02:54:19.421221 11008 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2f6e3bdb70e4dcebb69d5effeff7f83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2f6e3bdb70e4dcebb69d5effeff7f83. Latest consensus state: current_term: 1 leader_uuid: "c2f6e3bdb70e4dcebb69d5effeff7f83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2f6e3bdb70e4dcebb69d5effeff7f83" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 33071 } } }
I20260806 02:54:19.422850 11008 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2f6e3bdb70e4dcebb69d5effeff7f83 [sys.catalog]: This master's current role is: LEADER
I20260806 02:54:19.429700 11014 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:54:19.437521 11014 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:54:19.438722 10999 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 02:54:19.458547 11014 catalog_manager.cc:1347] Generated new cluster ID: 4089f2ab8b3f4574af10258a65a95704
I20260806 02:54:19.458807 11014 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:54:19.480160 11014 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:54:19.481686 11014 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:54:19.489038 11014 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2f6e3bdb70e4dcebb69d5effeff7f83: Generated new TSK 0
I20260806 02:54:19.490033 11014 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.211.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab.
Entry for principal kudu/127.25.211.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab.
I20260806 02:54:20.334812 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.211.148:54765/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.129:0
--local_ip_for_outbound_sockets=127.25.211.129
--webserver_interface=127.25.211.129
--webserver_port=0
--tserver_master_addrs=127.25.211.190:33071
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab
--principal=kudu/127.25.211.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:46835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:54:20 dist-test-slave-zj8q krb5kdc[10232](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785984860, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260806 02:54:20.879562 11061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:54:20.880082 11061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:54:20.880369 11061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:54:20.891537 11061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:54:20.891799 11061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:54:20.891980 11061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.129
I20260806 02:54:20.904069 11061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:46835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.25.211.148:54765/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab
--principal=kudu/127.25.211.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.211.129
--webserver_port=0
--tserver_master_addrs=127.25.211.190:33071
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.129
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:54:20.905587 11061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:54:20.907161 11061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:54:20.926880 11074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:20.928745 11073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:20.936537 11079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:54:20.936827 11061 server_base.cc:1034] running on GCE node
I20260806 02:54:20.937731 11061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:54:20.938983 11061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:54:20.940300 11061 hybrid_clock.cc:648] HybridClock initialized: now 1785984860940178 us; error 85 us; skew 500 ppm
I20260806 02:54:20.940687 11061 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:54:20 dist-test-slave-zj8q krb5kdc[10232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984860, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:54:20.945086 11061 init.cc:377] Logged in from keytab as kudu/127.25.211.129@KRBTEST.COM (short username kudu)
I20260806 02:54:20.952435 11061 webserver.cc:460] Webserver started at http://127.25.211.129:35213/ using document root <none> and password file <none>
I20260806 02:54:20.953824 11061 fs_manager.cc:359] Metadata directory not provided
I20260806 02:54:20.954181 11061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:54:20.954780 11061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 02:54:20 dist-test-slave-zj8q krb5kdc[10232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984860, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.129@KRBTEST.COM for HTTP/127.25.211.148@KRBTEST.COM
I20260806 02:54:20.980525 11061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ts-0/data/instance:
uuid: "4bebec71961e45e7b7570d426e0cd61c"
format_stamp: "Formatted at 2026-08-06 02:54:20 on dist-test-slave-zj8q"
server_key: "0dcaa9dc194cfc02b1e5fee0816ce85e"
server_key_iv: "2e548d7e86083e1699ed55e836baae25"
server_key_version: "kuduclusterkey@0"
I20260806 02:54:20.981859 11061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ts-0/wal/instance:
uuid: "4bebec71961e45e7b7570d426e0cd61c"
format_stamp: "Formatted at 2026-08-06 02:54:20 on dist-test-slave-zj8q"
server_key: "0dcaa9dc194cfc02b1e5fee0816ce85e"
server_key_iv: "2e548d7e86083e1699ed55e836baae25"
server_key_version: "kuduclusterkey@0"
I20260806 02:54:20.990558 11061 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.003s sys 0.004s
I20260806 02:54:21.010905 11089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:21.012811 11061 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260806 02:54:21.013011 11061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ts-0/wal
uuid: "4bebec71961e45e7b7570d426e0cd61c"
format_stamp: "Formatted at 2026-08-06 02:54:20 on dist-test-slave-zj8q"
server_key: "0dcaa9dc194cfc02b1e5fee0816ce85e"
server_key_iv: "2e548d7e86083e1699ed55e836baae25"
server_key_version: "kuduclusterkey@0"
I20260806 02:54:21.013391 11061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:54:21.035914 11061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:54:21.054548 11061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:54:21.055088 11061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:54:21.065610 11061 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:54:21.079339 11061 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:54:21.079635 11061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260806 02:54:21.079893 11061 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:54:21.080098 11061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 02:54:21 dist-test-slave-zj8q krb5kdc[10232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984860, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.190@KRBTEST.COM for HTTP/127.25.211.148@KRBTEST.COM
I20260806 02:54:21.660059 11061 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.129:43433
I20260806 02:54:21.660110 11257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.129:43433 every 8 connection(s)
I20260806 02:54:21.662266 11061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
I20260806 02:54:21.663939 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 11061
I20260806 02:54:21.664153 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ts-0/wal/instance
I20260806 02:54:21.665233 26446 external_mini_cluster.cc:1428] Setting key 27e083f63366d6289bcfd4caab46c274
Aug 06 02:54:21 dist-test-slave-zj8q krb5kdc[10232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984860, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.129@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:54:21.705328 11262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:21.666568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:33071 (local address 127.25.211.129:44957)
0806 02:54:21.675657 (+ 9089us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:21.675834 (+ 177us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:21.681068 (+ 5234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:21.682518 (+ 1450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:21.682535 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:21.683388 (+ 853us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:54:21.684369 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:21.684400 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:21.686124 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:21.686129 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:21.689289 (+ 3160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:21.689310 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:21.691022 (+ 1712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:21.691032 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:21.691372 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:21.692252 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:54:21.692287 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:54:21.695605 (+ 3318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:54:21.700377 (+ 4772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:54:21.700387 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:54:21.700406 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:54:21.701030 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:54:21.701528 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:54:21.701532 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:54:21.701536 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:54:21.701809 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:54:21.702591 (+ 782us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:54:21.702600 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:54:21.703084 (+ 484us) client_negotiation.cc:769] Sending connection context
0806 02:54:21.703399 (+ 315us) client_negotiation.cc:240] Negotiation successful
0806 02:54:21.703935 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8666,"thread_start_us":11491,"threads_started":1}
I20260806 02:54:21.706523 11261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:21.666283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:44957 (local address 127.25.211.190:33071)
0806 02:54:21.679807 (+ 13524us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:21.679814 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:21.679842 (+ 28us) server_negotiation.cc:407] Connection header received
0806 02:54:21.681528 (+ 1686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:21.681535 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:21.681641 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:21.681734 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:54:21.684605 (+ 2871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:21.685890 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:21.689530 (+ 3640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:21.690473 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:21.690588 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:21.695922 (+ 5334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:54:21.695929 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:54:21.695934 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:54:21.695975 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:54:21.700190 (+ 4215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:21.701202 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:21.701208 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:21.701211 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:21.701383 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:21.701951 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:21.701956 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:21.701958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:21.702378 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:54:21.702517 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:21.705884 (+ 3367us) server_negotiation.cc:299] Negotiation successful
0806 02:54:21.706124 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13375,"thread_start_us":15875,"threads_started":1}
I20260806 02:54:21.708096 11259 heartbeater.cc:347] Connected to a master server at 127.25.211.190:33071
I20260806 02:54:21.708516 11259 heartbeater.cc:464] Registering TS with master...
I20260806 02:54:21.709795 11259 heartbeater.cc:511] Master 127.25.211.190:33071 requested a full tablet report, sending...
I20260806 02:54:21.712885 10945 ts_manager.cc:194] Registered new tserver with Master: 4bebec71961e45e7b7570d426e0cd61c (127.25.211.129:43433)
I20260806 02:54:21.714996 10945 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.211.129@KRBTEST.COM'} at 127.25.211.129:44957
WARNING: no policy specified for kudu/127.25.211.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab.
Entry for principal kudu/127.25.211.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab.
I20260806 02:54:21.777729 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.211.148:54765/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.130:0
--local_ip_for_outbound_sockets=127.25.211.130
--webserver_interface=127.25.211.130
--webserver_port=0
--tserver_master_addrs=127.25.211.190:33071
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab
--principal=kudu/127.25.211.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:46835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:54:21 dist-test-slave-zj8q krb5kdc[10232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984860, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.190@KRBTEST.COM for HTTP/127.25.211.148@KRBTEST.COM
W20260806 02:54:22.362408 11267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:54:22.362848 11267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:54:22.363000 11267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:54:22.372073 11267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:54:22.372279 11267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:54:22.372491 11267 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.130
I20260806 02:54:22.384732 11267 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:46835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.25.211.148:54765/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab
--principal=kudu/127.25.211.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.211.130
--webserver_port=0
--tserver_master_addrs=127.25.211.190:33071
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.130
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:54:22.386459 11267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:54:22.388206 11267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:54:22.406755 11284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:22.406838 11285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:22.406849 11288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:54:22.408815 11267 server_base.cc:1034] running on GCE node
I20260806 02:54:22.409696 11267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:54:22.411556 11267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:54:22.412845 11267 hybrid_clock.cc:648] HybridClock initialized: now 1785984862412774 us; error 55 us; skew 500 ppm
I20260806 02:54:22.413396 11267 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:54:22 dist-test-slave-zj8q krb5kdc[10232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984862, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:54:22.418372 11267 init.cc:377] Logged in from keytab as kudu/127.25.211.130@KRBTEST.COM (short username kudu)
I20260806 02:54:22.420436 11267 webserver.cc:460] Webserver started at http://127.25.211.130:37659/ using document root <none> and password file <none>
I20260806 02:54:22.421474 11267 fs_manager.cc:359] Metadata directory not provided
I20260806 02:54:22.421749 11267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:54:22.422286 11267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 02:54:22 dist-test-slave-zj8q krb5kdc[10232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984862, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.130@KRBTEST.COM for HTTP/127.25.211.148@KRBTEST.COM
I20260806 02:54:22.442384 11267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ts-1/data/instance:
uuid: "63328cf8c6e4496d9f39a49bb8cb623e"
format_stamp: "Formatted at 2026-08-06 02:54:22 on dist-test-slave-zj8q"
server_key: "5c0dbdcaf8e83c319d6fa232445504ec"
server_key_iv: "f3f1cf80dd9ded269f043308beaf227f"
server_key_version: "kuduclusterkey@0"
I20260806 02:54:22.444064 11267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ts-1/wal/instance:
uuid: "63328cf8c6e4496d9f39a49bb8cb623e"
format_stamp: "Formatted at 2026-08-06 02:54:22 on dist-test-slave-zj8q"
server_key: "5c0dbdcaf8e83c319d6fa232445504ec"
server_key_iv: "f3f1cf80dd9ded269f043308beaf227f"
server_key_version: "kuduclusterkey@0"
I20260806 02:54:22.455106 11267 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.009s sys 0.000s
I20260806 02:54:22.479008 11295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:22.481175 11267 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.008s sys 0.000s
I20260806 02:54:22.481487 11267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ts-1/wal
uuid: "63328cf8c6e4496d9f39a49bb8cb623e"
format_stamp: "Formatted at 2026-08-06 02:54:22 on dist-test-slave-zj8q"
server_key: "5c0dbdcaf8e83c319d6fa232445504ec"
server_key_iv: "f3f1cf80dd9ded269f043308beaf227f"
server_key_version: "kuduclusterkey@0"
I20260806 02:54:22.481838 11267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:54:22.518765 11267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:54:22.525311 11267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:54:22.525796 11267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:54:22.527290 11267 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:54:22.529475 11267 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:54:22.529592 11267 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:22.529765 11267 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:54:22.529831 11267 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:22.601676 11267 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.130:35013
I20260806 02:54:22.601922 11453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.130:35013 every 8 connection(s)
I20260806 02:54:22.603353 11267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
I20260806 02:54:22.607962 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 11267
I20260806 02:54:22.608153 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ts-1/wal/instance
I20260806 02:54:22.608736 26446 external_mini_cluster.cc:1428] Setting key 762797e0d2c2161bb74588186e7f2ec6
Aug 06 02:54:22 dist-test-slave-zj8q krb5kdc[10232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984862, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.130@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:54:22.634626 11456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:22.606559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:53297 (local address 127.25.211.190:33071)
0806 02:54:22.607195 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:22.607201 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:22.610234 (+ 3033us) server_negotiation.cc:407] Connection header received
0806 02:54:22.611857 (+ 1623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:22.611862 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:22.611945 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:22.612050 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:54:22.614598 (+ 2548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:22.615612 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:22.618715 (+ 3103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:22.619600 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:22.619710 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:22.625249 (+ 5539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:54:22.625258 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:54:22.625263 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:54:22.625320 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:54:22.629881 (+ 4561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:22.631367 (+ 1486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:22.631374 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:22.631378 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:22.631550 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:22.632308 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:22.632315 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:22.632318 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:22.632751 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:54:22.632898 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:22.633969 (+ 1071us) server_negotiation.cc:299] Negotiation successful
0806 02:54:22.634258 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":386,"threads_started":1}
I20260806 02:54:22.636037 11458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:22.607487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:33071 (local address 127.25.211.130:53297)
0806 02:54:22.608702 (+ 1215us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:22.608758 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:22.611392 (+ 2634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:22.612534 (+ 1142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:22.612546 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:22.613319 (+ 773us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:54:22.614371 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:22.614403 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:22.615813 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:22.615818 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:22.618493 (+ 2675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:22.618524 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:22.620325 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:22.620336 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:22.620788 (+ 452us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:22.621792 (+ 1004us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:54:22.621824 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:54:22.624852 (+ 3028us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:54:22.630229 (+ 5377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:54:22.630244 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:54:22.630266 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:54:22.631170 (+ 904us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:54:22.631684 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:54:22.631689 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:54:22.631693 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:54:22.632140 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:54:22.632905 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:54:22.632915 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:54:22.633454 (+ 539us) client_negotiation.cc:769] Sending connection context
0806 02:54:22.633799 (+ 345us) client_negotiation.cc:240] Negotiation successful
0806 02:54:22.634514 (+ 715us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":876,"thread_start_us":430,"threads_started":1}
I20260806 02:54:22.638021 11454 heartbeater.cc:347] Connected to a master server at 127.25.211.190:33071
I20260806 02:54:22.638525 11454 heartbeater.cc:464] Registering TS with master...
I20260806 02:54:22.639653 11454 heartbeater.cc:511] Master 127.25.211.190:33071 requested a full tablet report, sending...
I20260806 02:54:22.642241 10945 ts_manager.cc:194] Registered new tserver with Master: 63328cf8c6e4496d9f39a49bb8cb623e (127.25.211.130:35013)
I20260806 02:54:22.643635 10945 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.211.130@KRBTEST.COM'} at 127.25.211.130:53297
WARNING: no policy specified for kudu/127.25.211.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab.
Entry for principal kudu/127.25.211.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab.
I20260806 02:54:22.690905 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.211.148:54765/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.131:0
--local_ip_for_outbound_sockets=127.25.211.131
--webserver_interface=127.25.211.131
--webserver_port=0
--tserver_master_addrs=127.25.211.190:33071
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab
--principal=kudu/127.25.211.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:46835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:54:22.719548 11259 heartbeater.cc:503] Master 127.25.211.190:33071 was elected leader, sending a full tablet report...
W20260806 02:54:23.159888 11461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:54:23.160202 11461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:54:23.160302 11461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:54:23.169198 11461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:54:23.169401 11461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:54:23.169621 11461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.131
I20260806 02:54:23.180614 11461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:46835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.25.211.148:54765/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab
--principal=kudu/127.25.211.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.211.131
--webserver_port=0
--tserver_master_addrs=127.25.211.190:33071
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.131
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:54:23.181968 11461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:54:23.183308 11461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:54:23.199632 11466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:23.199596 11467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:54:23.199713 11472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:54:23.201360 11461 server_base.cc:1034] running on GCE node
I20260806 02:54:23.202011 11461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:54:23.203088 11461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:54:23.204321 11461 hybrid_clock.cc:648] HybridClock initialized: now 1785984863204271 us; error 44 us; skew 500 ppm
I20260806 02:54:23.204706 11461 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:54:23 dist-test-slave-zj8q krb5kdc[10232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984863, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:54:23.208598 11461 init.cc:377] Logged in from keytab as kudu/127.25.211.131@KRBTEST.COM (short username kudu)
I20260806 02:54:23.210425 11461 webserver.cc:460] Webserver started at http://127.25.211.131:33411/ using document root <none> and password file <none>
I20260806 02:54:23.211267 11461 fs_manager.cc:359] Metadata directory not provided
I20260806 02:54:23.211398 11461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:54:23.211720 11461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 02:54:23 dist-test-slave-zj8q krb5kdc[10232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984863, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.131@KRBTEST.COM for HTTP/127.25.211.148@KRBTEST.COM
I20260806 02:54:23.227715 11461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ts-2/data/instance:
uuid: "2832c55a6ad84caab75d0152e0de0cbc"
format_stamp: "Formatted at 2026-08-06 02:54:23 on dist-test-slave-zj8q"
server_key: "b641a834f3d69c7c4ac4c10eb88b2a75"
server_key_iv: "ec6ec930393c0baa99202466a657179a"
server_key_version: "kuduclusterkey@0"
I20260806 02:54:23.228627 11461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ts-2/wal/instance:
uuid: "2832c55a6ad84caab75d0152e0de0cbc"
format_stamp: "Formatted at 2026-08-06 02:54:23 on dist-test-slave-zj8q"
server_key: "b641a834f3d69c7c4ac4c10eb88b2a75"
server_key_iv: "ec6ec930393c0baa99202466a657179a"
server_key_version: "kuduclusterkey@0"
I20260806 02:54:23.235252 11461 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260806 02:54:23.251539 11481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:23.253485 11461 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260806 02:54:23.253691 11461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ts-2/wal
uuid: "2832c55a6ad84caab75d0152e0de0cbc"
format_stamp: "Formatted at 2026-08-06 02:54:23 on dist-test-slave-zj8q"
server_key: "b641a834f3d69c7c4ac4c10eb88b2a75"
server_key_iv: "ec6ec930393c0baa99202466a657179a"
server_key_version: "kuduclusterkey@0"
I20260806 02:54:23.253944 11461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:54:23.281771 11461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:54:23.286845 11461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:54:23.287210 11461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:54:23.288390 11461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:54:23.290172 11461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:54:23.290323 11461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260806 02:54:23.290459 11461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:54:23.290516 11461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:54:23.335487 11461 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.131:34253
I20260806 02:54:23.335631 11646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.131:34253 every 8 connection(s)
I20260806 02:54:23.336864 11461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
I20260806 02:54:23.345017 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 11461
I20260806 02:54:23.345235 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ts-2/wal/instance
I20260806 02:54:23.345790 26446 external_mini_cluster.cc:1428] Setting key 9c6b821ed9fcb65660eeeb2492a1005f
Aug 06 02:54:23 dist-test-slave-zj8q krb5kdc[10232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984863, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.131@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:54:23.362931 11652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:23.339897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:60105 (local address 127.25.211.190:33071)
0806 02:54:23.340487 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:23.340492 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:23.341623 (+ 1131us) server_negotiation.cc:407] Connection header received
0806 02:54:23.343536 (+ 1913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:23.343542 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:23.343708 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:23.343816 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:54:23.345982 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:23.346999 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:23.348862 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:23.349405 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:23.349524 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:23.353768 (+ 4244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:54:23.353775 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:54:23.353780 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:54:23.353829 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:54:23.358215 (+ 4386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:23.359227 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:23.359233 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:23.359237 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:23.359411 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:23.360242 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:23.360248 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:23.360252 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:23.360632 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:54:23.360831 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:23.362004 (+ 1173us) server_negotiation.cc:299] Negotiation successful
0806 02:54:23.362482 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":317,"threads_started":1}
I20260806 02:54:23.363577 11653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:23.340464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:33071 (local address 127.25.211.131:60105)
0806 02:54:23.341425 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:23.341471 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:23.343244 (+ 1773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:23.344088 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:23.344096 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:23.344696 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:54:23.345712 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:23.345787 (+ 75us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:23.347168 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:23.347173 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:23.348695 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:23.348710 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:23.349650 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:23.349658 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:23.349984 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:23.350833 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:54:23.350860 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:54:23.353465 (+ 2605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:54:23.358418 (+ 4953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:54:23.358427 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:54:23.358442 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:54:23.359092 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:54:23.359529 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:54:23.359532 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:54:23.359535 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:54:23.359856 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:54:23.360911 (+ 1055us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:54:23.360921 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:54:23.361496 (+ 575us) client_negotiation.cc:769] Sending connection context
0806 02:54:23.361847 (+ 351us) client_negotiation.cc:240] Negotiation successful
0806 02:54:23.362368 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":721,"thread_start_us":476,"threads_started":1}
I20260806 02:54:23.364939 11648 heartbeater.cc:347] Connected to a master server at 127.25.211.190:33071
I20260806 02:54:23.365278 11648 heartbeater.cc:464] Registering TS with master...
I20260806 02:54:23.366135 11648 heartbeater.cc:511] Master 127.25.211.190:33071 requested a full tablet report, sending...
I20260806 02:54:23.367729 10945 ts_manager.cc:194] Registered new tserver with Master: 2832c55a6ad84caab75d0152e0de0cbc (127.25.211.131:34253)
I20260806 02:54:23.368569 10945 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.211.131@KRBTEST.COM'} at 127.25.211.131:60105
I20260806 02:54:23.375063 26446 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:54:23.394441 11652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:23.380339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37736 (local address 127.25.211.190:33071)
0806 02:54:23.380515 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:23.380520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:23.380937 (+ 417us) server_negotiation.cc:407] Connection header received
0806 02:54:23.381088 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:23.381092 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:23.381174 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:23.381267 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:54:23.382874 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:23.383671 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:23.385272 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:23.385798 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:23.385906 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:23.388479 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:54:23.388484 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:54:23.388488 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:54:23.388526 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:54:23.391457 (+ 2931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:23.392244 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:23.392249 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:23.392251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:23.392390 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:23.392791 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:23.392795 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:23.392797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:23.393093 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:54:23.393347 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:23.393724 (+ 377us) server_negotiation.cc:299] Negotiation successful
0806 02:54:23.393979 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 02:54:23.398904 10945 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37736:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:54:23.407330 10945 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:54:23.440059 11675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:23.421435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:43433 (local address 127.0.0.1:52450)
0806 02:54:23.424245 (+ 2810us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:23.424330 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:23.424676 (+ 346us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:23.425908 (+ 1232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:23.425917 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:23.429445 (+ 3528us) spinlock_profiling.cc:243] Waited 2.79 ms on lock 0x61200002c7a8. stack: 00007f950865166e 00007f9508651430 00007f95076d85ed 00000000008ea064 00000000008ea02d 00000000008e99fa 00007f95086470fd 00007f9508646d1e 00007f951b5503cd 00007f951b55036c 00007f951b51cb3b 00007f950f0588e9 00007f950f054e45 00007f950f0dcdca 00007f950f0daf68 00007f950f10e82f
0806 02:54:23.429488 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:54:23.430490 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:23.430512 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:23.433353 (+ 2841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:23.433359 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:23.436954 (+ 3595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:23.436978 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:23.438804 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:23.438811 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:23.439020 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:23.439079 (+ 59us) client_negotiation.cc:769] Sending connection context
0806 02:54:23.439233 (+ 154us) client_negotiation.cc:240] Negotiation successful
0806 02:54:23.439462 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2395,"mutex_wait_us":133,"spinlock_wait_cycles":6422272,"thread_start_us":2786,"threads_started":1}
I20260806 02:54:23.441090 11671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:23.421661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52450 (local address 127.25.211.129:43433)
0806 02:54:23.422739 (+ 1078us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:23.422748 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:23.424471 (+ 1723us) server_negotiation.cc:407] Connection header received
0806 02:54:23.424987 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:23.424996 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:23.425329 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:23.425568 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:54:23.430793 (+ 5225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:23.433151 (+ 2358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:23.437206 (+ 4055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:23.438642 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:23.438848 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:23.439053 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:23.440386 (+ 1333us) server_negotiation.cc:299] Negotiation successful
0806 02:54:23.440672 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":907,"thread_start_us":737,"threads_started":1}
I20260806 02:54:23.441128 11676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:23.424501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.130:35013 (local address 127.0.0.1:50660)
0806 02:54:23.431016 (+ 6515us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:23.431036 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:23.431153 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:23.432353 (+ 1200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:23.432445 (+ 92us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:23.432484 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:54:23.432956 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:23.432970 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:23.435577 (+ 2607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:23.435583 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:23.438011 (+ 2428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:23.438028 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:23.440048 (+ 2020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:23.440056 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:23.440223 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:23.440256 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 02:54:23.440345 (+ 89us) client_negotiation.cc:240] Negotiation successful
0806 02:54:23.440478 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6380,"mutex_wait_us":25,"thread_start_us":6203,"threads_started":1}
I20260806 02:54:23.442117 11677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:23.424984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50660 (local address 127.25.211.130:35013)
0806 02:54:23.431577 (+ 6593us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:23.431586 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:23.431617 (+ 31us) server_negotiation.cc:407] Connection header received
0806 02:54:23.431710 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:23.431718 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:23.431979 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:23.432215 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:54:23.433142 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:23.435419 (+ 2277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:23.438251 (+ 2832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:23.439501 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:23.439679 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:23.439854 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:23.441380 (+ 1526us) server_negotiation.cc:299] Negotiation successful
0806 02:54:23.441616 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6400,"thread_start_us":6526,"threads_started":1}
I20260806 02:54:23.442199 11669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:23.422888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.131:34253 (local address 127.0.0.1:37764)
0806 02:54:23.424523 (+ 1635us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:23.424539 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:23.428573 (+ 4034us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:23.431199 (+ 2626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:23.431205 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:23.431236 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:54:23.431614 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:23.431630 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:23.433374 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:23.433379 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:23.436983 (+ 3604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:23.436997 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:23.441193 (+ 4196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:23.441200 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:23.441382 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:23.441414 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 02:54:23.441499 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 02:54:23.441637 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1521,"mutex_wait_us":181,"thread_start_us":682,"threads_started":1}
I20260806 02:54:23.442199 11673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:23.422350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37764 (local address 127.25.211.131:34253)
0806 02:54:23.423446 (+ 1096us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:23.423456 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:23.424651 (+ 1195us) server_negotiation.cc:407] Connection header received
0806 02:54:23.430621 (+ 5970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:23.430634 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:23.430953 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:23.431192 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:54:23.431813 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:23.433240 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:23.439196 (+ 5956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:23.440672 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:23.440855 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:23.441018 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:23.441554 (+ 536us) server_negotiation.cc:299] Negotiation successful
0806 02:54:23.441766 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":906,"thread_start_us":687,"threads_started":1}
I20260806 02:54:23.445792 11160 tablet_service.cc:1467] Processing CreateTablet for tablet 4b07a941bab2418dbc898429feef2339 (DEFAULT_TABLE table=test-table [id=19116a9e23104f44a0c97d3cd35e9df4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:54:23.445911 11361 tablet_service.cc:1467] Processing CreateTablet for tablet 4b07a941bab2418dbc898429feef2339 (DEFAULT_TABLE table=test-table [id=19116a9e23104f44a0c97d3cd35e9df4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:54:23.445994 11549 tablet_service.cc:1467] Processing CreateTablet for tablet 4b07a941bab2418dbc898429feef2339 (DEFAULT_TABLE table=test-table [id=19116a9e23104f44a0c97d3cd35e9df4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:54:23.447551 11160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b07a941bab2418dbc898429feef2339. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:54:23.447588 11549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b07a941bab2418dbc898429feef2339. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:54:23.447597 11361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b07a941bab2418dbc898429feef2339. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:54:23.460595 11678 tablet_bootstrap.cc:492] T 4b07a941bab2418dbc898429feef2339 P 2832c55a6ad84caab75d0152e0de0cbc: Bootstrap starting.
I20260806 02:54:23.462984 11679 tablet_bootstrap.cc:492] T 4b07a941bab2418dbc898429feef2339 P 63328cf8c6e4496d9f39a49bb8cb623e: Bootstrap starting.
I20260806 02:54:23.464622 11680 tablet_bootstrap.cc:492] T 4b07a941bab2418dbc898429feef2339 P 4bebec71961e45e7b7570d426e0cd61c: Bootstrap starting.
I20260806 02:54:23.467311 11678 tablet_bootstrap.cc:654] T 4b07a941bab2418dbc898429feef2339 P 2832c55a6ad84caab75d0152e0de0cbc: Neither blocks nor log segments found. Creating new log.
I20260806 02:54:23.467310 11679 tablet_bootstrap.cc:654] T 4b07a941bab2418dbc898429feef2339 P 63328cf8c6e4496d9f39a49bb8cb623e: Neither blocks nor log segments found. Creating new log.
I20260806 02:54:23.468518 11680 tablet_bootstrap.cc:654] T 4b07a941bab2418dbc898429feef2339 P 4bebec71961e45e7b7570d426e0cd61c: Neither blocks nor log segments found. Creating new log.
I20260806 02:54:23.468924 11678 log.cc:824] T 4b07a941bab2418dbc898429feef2339 P 2832c55a6ad84caab75d0152e0de0cbc: Log is configured to *not* fsync() on all Append() calls
I20260806 02:54:23.468924 11679 log.cc:824] T 4b07a941bab2418dbc898429feef2339 P 63328cf8c6e4496d9f39a49bb8cb623e: Log is configured to *not* fsync() on all Append() calls
I20260806 02:54:23.469720 11680 log.cc:824] T 4b07a941bab2418dbc898429feef2339 P 4bebec71961e45e7b7570d426e0cd61c: Log is configured to *not* fsync() on all Append() calls
I20260806 02:54:23.472939 11678 tablet_bootstrap.cc:492] T 4b07a941bab2418dbc898429feef2339 P 2832c55a6ad84caab75d0152e0de0cbc: No bootstrap required, opened a new log
I20260806 02:54:23.473147 11679 tablet_bootstrap.cc:492] T 4b07a941bab2418dbc898429feef2339 P 63328cf8c6e4496d9f39a49bb8cb623e: No bootstrap required, opened a new log
I20260806 02:54:23.473243 11678 ts_tablet_manager.cc:1397] T 4b07a941bab2418dbc898429feef2339 P 2832c55a6ad84caab75d0152e0de0cbc: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20260806 02:54:23.473567 11679 ts_tablet_manager.cc:1397] T 4b07a941bab2418dbc898429feef2339 P 63328cf8c6e4496d9f39a49bb8cb623e: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260806 02:54:23.474206 11680 tablet_bootstrap.cc:492] T 4b07a941bab2418dbc898429feef2339 P 4bebec71961e45e7b7570d426e0cd61c: No bootstrap required, opened a new log
I20260806 02:54:23.474576 11680 ts_tablet_manager.cc:1397] T 4b07a941bab2418dbc898429feef2339 P 4bebec71961e45e7b7570d426e0cd61c: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.006s
I20260806 02:54:23.482537 11678 raft_consensus.cc:348] T 4b07a941bab2418dbc898429feef2339 P 2832c55a6ad84caab75d0152e0de0cbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bebec71961e45e7b7570d426e0cd61c" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 43433 } } peers { permanent_uuid: "2832c55a6ad84caab75d0152e0de0cbc" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 34253 } } peers { permanent_uuid: "63328cf8c6e4496d9f39a49bb8cb623e" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 35013 } }
I20260806 02:54:23.482972 11678 raft_consensus.cc:374] T 4b07a941bab2418dbc898429feef2339 P 2832c55a6ad84caab75d0152e0de0cbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:54:23.483086 11678 raft_consensus.cc:729] T 4b07a941bab2418dbc898429feef2339 P 2832c55a6ad84caab75d0152e0de0cbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2832c55a6ad84caab75d0152e0de0cbc, State: Initialized, Role: FOLLOWER
I20260806 02:54:23.483757 11678 consensus_queue.cc:260] T 4b07a941bab2418dbc898429feef2339 P 2832c55a6ad84caab75d0152e0de0cbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bebec71961e45e7b7570d426e0cd61c" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 43433 } } peers { permanent_uuid: "2832c55a6ad84caab75d0152e0de0cbc" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 34253 } } peers { permanent_uuid: "63328cf8c6e4496d9f39a49bb8cb623e" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 35013 } }
I20260806 02:54:23.484997 11648 heartbeater.cc:503] Master 127.25.211.190:33071 was elected leader, sending a full tablet report...
I20260806 02:54:23.485426 11679 raft_consensus.cc:348] T 4b07a941bab2418dbc898429feef2339 P 63328cf8c6e4496d9f39a49bb8cb623e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bebec71961e45e7b7570d426e0cd61c" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 43433 } } peers { permanent_uuid: "2832c55a6ad84caab75d0152e0de0cbc" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 34253 } } peers { permanent_uuid: "63328cf8c6e4496d9f39a49bb8cb623e" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 35013 } }
I20260806 02:54:23.485803 11678 ts_tablet_manager.cc:1428] T 4b07a941bab2418dbc898429feef2339 P 2832c55a6ad84caab75d0152e0de0cbc: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260806 02:54:23.485987 11679 raft_consensus.cc:374] T 4b07a941bab2418dbc898429feef2339 P 63328cf8c6e4496d9f39a49bb8cb623e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:54:23.485746 11680 raft_consensus.cc:348] T 4b07a941bab2418dbc898429feef2339 P 4bebec71961e45e7b7570d426e0cd61c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bebec71961e45e7b7570d426e0cd61c" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 43433 } } peers { permanent_uuid: "2832c55a6ad84caab75d0152e0de0cbc" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 34253 } } peers { permanent_uuid: "63328cf8c6e4496d9f39a49bb8cb623e" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 35013 } }
I20260806 02:54:23.486235 11680 raft_consensus.cc:374] T 4b07a941bab2418dbc898429feef2339 P 4bebec71961e45e7b7570d426e0cd61c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:54:23.486233 11679 raft_consensus.cc:729] T 4b07a941bab2418dbc898429feef2339 P 63328cf8c6e4496d9f39a49bb8cb623e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63328cf8c6e4496d9f39a49bb8cb623e, State: Initialized, Role: FOLLOWER
I20260806 02:54:23.486351 11680 raft_consensus.cc:729] T 4b07a941bab2418dbc898429feef2339 P 4bebec71961e45e7b7570d426e0cd61c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bebec71961e45e7b7570d426e0cd61c, State: Initialized, Role: FOLLOWER
I20260806 02:54:23.487015 11680 consensus_queue.cc:260] T 4b07a941bab2418dbc898429feef2339 P 4bebec71961e45e7b7570d426e0cd61c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bebec71961e45e7b7570d426e0cd61c" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 43433 } } peers { permanent_uuid: "2832c55a6ad84caab75d0152e0de0cbc" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 34253 } } peers { permanent_uuid: "63328cf8c6e4496d9f39a49bb8cb623e" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 35013 } }
I20260806 02:54:23.487015 11679 consensus_queue.cc:260] T 4b07a941bab2418dbc898429feef2339 P 63328cf8c6e4496d9f39a49bb8cb623e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bebec71961e45e7b7570d426e0cd61c" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 43433 } } peers { permanent_uuid: "2832c55a6ad84caab75d0152e0de0cbc" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 34253 } } peers { permanent_uuid: "63328cf8c6e4496d9f39a49bb8cb623e" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 35013 } }
I20260806 02:54:23.489553 11454 heartbeater.cc:503] Master 127.25.211.190:33071 was elected leader, sending a full tablet report...
I20260806 02:54:23.489939 11679 ts_tablet_manager.cc:1428] T 4b07a941bab2418dbc898429feef2339 P 63328cf8c6e4496d9f39a49bb8cb623e: Time spent starting tablet: real 0.016s user 0.006s sys 0.010s
I20260806 02:54:23.489930 11680 ts_tablet_manager.cc:1428] T 4b07a941bab2418dbc898429feef2339 P 4bebec71961e45e7b7570d426e0cd61c: Time spent starting tablet: real 0.015s user 0.008s sys 0.006s
W20260806 02:54:23.589010 11649 tablet.cc:2369] T 4b07a941bab2418dbc898429feef2339 P 2832c55a6ad84caab75d0152e0de0cbc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:54:23.589172 11689 raft_consensus.cc:482] T 4b07a941bab2418dbc898429feef2339 P 63328cf8c6e4496d9f39a49bb8cb623e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:54:23.589506 11689 raft_consensus.cc:504] T 4b07a941bab2418dbc898429feef2339 P 63328cf8c6e4496d9f39a49bb8cb623e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bebec71961e45e7b7570d426e0cd61c" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 43433 } } peers { permanent_uuid: "2832c55a6ad84caab75d0152e0de0cbc" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 34253 } } peers { permanent_uuid: "63328cf8c6e4496d9f39a49bb8cb623e" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 35013 } }
I20260806 02:54:23.591481 11689 leader_election.cc:288] T 4b07a941bab2418dbc898429feef2339 P 63328cf8c6e4496d9f39a49bb8cb623e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bebec71961e45e7b7570d426e0cd61c (127.25.211.129:43433), 2832c55a6ad84caab75d0152e0de0cbc (127.25.211.131:34253)
I20260806 02:54:23.600281 11673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:23.592395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:46745 (local address 127.25.211.131:34253)
0806 02:54:23.592614 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:23.592621 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:23.592789 (+ 168us) server_negotiation.cc:407] Connection header received
0806 02:54:23.593047 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:23.593051 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:23.593110 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:23.593186 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:54:23.594300 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:23.595283 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:23.597582 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:23.599188 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:23.599402 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:23.599488 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:23.599883 (+ 395us) server_negotiation.cc:299] Negotiation successful
0806 02:54:23.600029 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 02:54:23.600385 11671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:23.592012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:51887 (local address 127.25.211.129:43433)
0806 02:54:23.592657 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:23.592664 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:23.593816 (+ 1152us) server_negotiation.cc:407] Connection header received
0806 02:54:23.594028 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:23.594032 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:23.594105 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:23.594263 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:54:23.594889 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:23.595827 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:23.598132 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:23.599366 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:23.599516 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:23.599573 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:23.599980 (+ 407us) server_negotiation.cc:299] Negotiation successful
0806 02:54:23.600118 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387}
I20260806 02:54:23.600988 11690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:23.591757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.131:34253 (local address 127.25.211.130:46745)
0806 02:54:23.592685 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:23.592704 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:23.592901 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:23.593244 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:23.593248 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:23.593283 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:54:23.594076 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:23.594088 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:23.595400 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:23.595403 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:23.597414 (+ 2011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:23.597429 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:23.599406 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:23.599411 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:23.599659 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:23.599698 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 02:54:23.599816 (+ 118us) client_negotiation.cc:240] Negotiation successful
0806 02:54:23.599979 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":777,"mutex_wait_us":38,"thread_start_us":877,"threads_started":1}
I20260806 02:54:23.601344 11691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:23.592085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:43433 (local address 127.25.211.130:51887)
0806 02:54:23.593629 (+ 1544us) negotiation.cc:107] Waiting for socket to connect
0806 02:54:23.593728 (+ 99us) client_negotiation.cc:174] Beginning negotiation
0806 02:54:23.593906 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:54:23.594275 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:54:23.594280 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:54:23.594306 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:54:23.594745 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:23.594755 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:23.595965 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:23.595970 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:23.597976 (+ 2006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:54:23.597987 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:54:23.599573 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:54:23.599579 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:54:23.599796 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:23.599826 (+ 30us) client_negotiation.cc:769] Sending connection context
0806 02:54:23.600582 (+ 756us) client_negotiation.cc:240] Negotiation successful
0806 02:54:23.600688 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1374,"mutex_wait_us":1,"thread_start_us":1081,"threads_started":1}
I20260806 02:54:23.602586 11578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b07a941bab2418dbc898429feef2339" candidate_uuid: "63328cf8c6e4496d9f39a49bb8cb623e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2832c55a6ad84caab75d0152e0de0cbc" is_pre_election: true
I20260806 02:54:23.602578 11196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b07a941bab2418dbc898429feef2339" candidate_uuid: "63328cf8c6e4496d9f39a49bb8cb623e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bebec71961e45e7b7570d426e0cd61c" is_pre_election: true
I20260806 02:54:23.603189 11578 raft_consensus.cc:2454] T 4b07a941bab2418dbc898429feef2339 P 2832c55a6ad84caab75d0152e0de0cbc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63328cf8c6e4496d9f39a49bb8cb623e in term 0.
I20260806 02:54:23.603189 11196 raft_consensus.cc:2454] T 4b07a941bab2418dbc898429feef2339 P 4bebec71961e45e7b7570d426e0cd61c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63328cf8c6e4496d9f39a49bb8cb623e in term 0.
I20260806 02:54:23.604086 11299 leader_election.cc:302] T 4b07a941bab2418dbc898429feef2339 P 63328cf8c6e4496d9f39a49bb8cb623e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2832c55a6ad84caab75d0152e0de0cbc, 63328cf8c6e4496d9f39a49bb8cb623e; no voters:
I20260806 02:54:23.604650 11689 raft_consensus.cc:2781] T 4b07a941bab2418dbc898429feef2339 P 63328cf8c6e4496d9f39a49bb8cb623e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:54:23.604877 11689 raft_consensus.cc:482] T 4b07a941bab2418dbc898429feef2339 P 63328cf8c6e4496d9f39a49bb8cb623e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:54:23.605002 11689 raft_consensus.cc:3037] T 4b07a941bab2418dbc898429feef2339 P 63328cf8c6e4496d9f39a49bb8cb623e [term 0 FOLLOWER]: Advancing to term 1
W20260806 02:54:23.606324 11455 tablet.cc:2369] T 4b07a941bab2418dbc898429feef2339 P 63328cf8c6e4496d9f39a49bb8cb623e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:54:23.608752 11689 raft_consensus.cc:504] T 4b07a941bab2418dbc898429feef2339 P 63328cf8c6e4496d9f39a49bb8cb623e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bebec71961e45e7b7570d426e0cd61c" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 43433 } } peers { permanent_uuid: "2832c55a6ad84caab75d0152e0de0cbc" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 34253 } } peers { permanent_uuid: "63328cf8c6e4496d9f39a49bb8cb623e" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 35013 } }
I20260806 02:54:23.609660 11689 leader_election.cc:288] T 4b07a941bab2418dbc898429feef2339 P 63328cf8c6e4496d9f39a49bb8cb623e [CANDIDATE]: Term 1 election: Requested vote from peers 4bebec71961e45e7b7570d426e0cd61c (127.25.211.129:43433), 2832c55a6ad84caab75d0152e0de0cbc (127.25.211.131:34253)
I20260806 02:54:23.609952 11196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b07a941bab2418dbc898429feef2339" candidate_uuid: "63328cf8c6e4496d9f39a49bb8cb623e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bebec71961e45e7b7570d426e0cd61c"
I20260806 02:54:23.610236 11196 raft_consensus.cc:3037] T 4b07a941bab2418dbc898429feef2339 P 4bebec71961e45e7b7570d426e0cd61c [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:54:23.612984 11196 raft_consensus.cc:2454] T 4b07a941bab2418dbc898429feef2339 P 4bebec71961e45e7b7570d426e0cd61c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63328cf8c6e4496d9f39a49bb8cb623e in term 1.
I20260806 02:54:23.613469 11298 leader_election.cc:302] T 4b07a941bab2418dbc898429feef2339 P 63328cf8c6e4496d9f39a49bb8cb623e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bebec71961e45e7b7570d426e0cd61c, 63328cf8c6e4496d9f39a49bb8cb623e; no voters:
I20260806 02:54:23.613790 11689 raft_consensus.cc:2781] T 4b07a941bab2418dbc898429feef2339 P 63328cf8c6e4496d9f39a49bb8cb623e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:54:23.613909 11578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b07a941bab2418dbc898429feef2339" candidate_uuid: "63328cf8c6e4496d9f39a49bb8cb623e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2832c55a6ad84caab75d0152e0de0cbc"
I20260806 02:54:23.614261 11578 raft_consensus.cc:3037] T 4b07a941bab2418dbc898429feef2339 P 2832c55a6ad84caab75d0152e0de0cbc [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:54:23.614305 11689 raft_consensus.cc:686] T 4b07a941bab2418dbc898429feef2339 P 63328cf8c6e4496d9f39a49bb8cb623e [term 1 LEADER]: Becoming Leader. State: Replica: 63328cf8c6e4496d9f39a49bb8cb623e, State: Running, Role: LEADER
I20260806 02:54:23.614948 11689 consensus_queue.cc:237] T 4b07a941bab2418dbc898429feef2339 P 63328cf8c6e4496d9f39a49bb8cb623e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bebec71961e45e7b7570d426e0cd61c" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 43433 } } peers { permanent_uuid: "2832c55a6ad84caab75d0152e0de0cbc" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 34253 } } peers { permanent_uuid: "63328cf8c6e4496d9f39a49bb8cb623e" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 35013 } }
I20260806 02:54:23.617463 11578 raft_consensus.cc:2454] T 4b07a941bab2418dbc898429feef2339 P 2832c55a6ad84caab75d0152e0de0cbc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63328cf8c6e4496d9f39a49bb8cb623e in term 1.
I20260806 02:54:23.620051 10942 catalog_manager.cc:5495] T 4b07a941bab2418dbc898429feef2339 P 63328cf8c6e4496d9f39a49bb8cb623e reported cstate change: term changed from 0 to 1, leader changed from <none> to 63328cf8c6e4496d9f39a49bb8cb623e (127.25.211.130). New cstate: current_term: 1 leader_uuid: "63328cf8c6e4496d9f39a49bb8cb623e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bebec71961e45e7b7570d426e0cd61c" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 43433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2832c55a6ad84caab75d0152e0de0cbc" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 34253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63328cf8c6e4496d9f39a49bb8cb623e" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 35013 } health_report { overall_health: HEALTHY } } }
I20260806 02:54:23.677505 11677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:23.671360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50676 (local address 127.25.211.130:35013)
0806 02:54:23.671573 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:23.671579 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:23.671598 (+ 19us) server_negotiation.cc:407] Connection header received
0806 02:54:23.671694 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:23.671698 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:23.671784 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:23.671920 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:54:23.672321 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:23.673390 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:23.675098 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:23.675522 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:23.675640 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:23.676237 (+ 597us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:54:23.676807 (+ 570us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:54:23.676954 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:23.677058 (+ 104us) server_negotiation.cc:299] Negotiation successful
0806 02:54:23.677148 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260806 02:54:23.684988 11260 tablet.cc:2369] T 4b07a941bab2418dbc898429feef2339 P 4bebec71961e45e7b7570d426e0cd61c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:54:23.686540 11196 raft_consensus.cc:1261] T 4b07a941bab2418dbc898429feef2339 P 4bebec71961e45e7b7570d426e0cd61c [term 1 FOLLOWER]: Refusing update from remote peer 63328cf8c6e4496d9f39a49bb8cb623e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:54:23.686542 11578 raft_consensus.cc:1261] T 4b07a941bab2418dbc898429feef2339 P 2832c55a6ad84caab75d0152e0de0cbc [term 1 FOLLOWER]: Refusing update from remote peer 63328cf8c6e4496d9f39a49bb8cb623e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:54:23.687947 11689 consensus_queue.cc:1035] T 4b07a941bab2418dbc898429feef2339 P 63328cf8c6e4496d9f39a49bb8cb623e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bebec71961e45e7b7570d426e0cd61c" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 43433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:54:23.688328 11692 consensus_queue.cc:1035] T 4b07a941bab2418dbc898429feef2339 P 63328cf8c6e4496d9f39a49bb8cb623e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2832c55a6ad84caab75d0152e0de0cbc" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 34253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:54:23.701421 11695 mvcc.cc:205] Tried to move back new op lower bound from 7315394001648865280 to 7315394001371336704. Current Snapshot: MvccSnapshot[applied={T|T < 7315394001648865280}]
I20260806 02:54:23.703402 11698 mvcc.cc:205] Tried to move back new op lower bound from 7315394001648865280 to 7315394001371336704. Current Snapshot: MvccSnapshot[applied={T|T < 7315394001648865280}]
I20260806 02:54:23.717005 10941 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:54:23.717336 10941 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:54:23.721732 10941 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c2f6e3bdb70e4dcebb69d5effeff7f83: Sending DeleteTablet for 3 replicas of tablet 4b07a941bab2418dbc898429feef2339
I20260806 02:54:23.723295 11160 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b07a941bab2418dbc898429feef2339 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:54:23 UTC) from {username='kudu', principal='kudu/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:52450
I20260806 02:54:23.723409 11549 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b07a941bab2418dbc898429feef2339 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:54:23 UTC) from {username='kudu', principal='kudu/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:37764
I20260806 02:54:23.723595 11361 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b07a941bab2418dbc898429feef2339 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:54:23 UTC) from {username='kudu', principal='kudu/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:50660
I20260806 02:54:23.726438 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 11061
I20260806 02:54:23.735771 11710 tablet_replica.cc:330] T 4b07a941bab2418dbc898429feef2339 P 2832c55a6ad84caab75d0152e0de0cbc: stopping tablet replica
I20260806 02:54:23.736347 11710 raft_consensus.cc:2229] T 4b07a941bab2418dbc898429feef2339 P 2832c55a6ad84caab75d0152e0de0cbc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:54:23.736956 11710 raft_consensus.cc:2258] T 4b07a941bab2418dbc898429feef2339 P 2832c55a6ad84caab75d0152e0de0cbc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:54:23.739236 11711 tablet_replica.cc:330] T 4b07a941bab2418dbc898429feef2339 P 63328cf8c6e4496d9f39a49bb8cb623e: stopping tablet replica
I20260806 02:54:23.739656 11710 ts_tablet_manager.cc:1905] T 4b07a941bab2418dbc898429feef2339 P 2832c55a6ad84caab75d0152e0de0cbc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:54:23.743866 11711 raft_consensus.cc:2229] T 4b07a941bab2418dbc898429feef2339 P 63328cf8c6e4496d9f39a49bb8cb623e [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:54:23.745036 11709 tablet_replica.cc:330] T 4b07a941bab2418dbc898429feef2339 P 4bebec71961e45e7b7570d426e0cd61c: stopping tablet replica
I20260806 02:54:23.745411 11709 raft_consensus.cc:2229] T 4b07a941bab2418dbc898429feef2339 P 4bebec71961e45e7b7570d426e0cd61c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:54:23.745991 11709 raft_consensus.cc:2258] T 4b07a941bab2418dbc898429feef2339 P 4bebec71961e45e7b7570d426e0cd61c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:54:23.746737 11711 raft_consensus.cc:2258] T 4b07a941bab2418dbc898429feef2339 P 63328cf8c6e4496d9f39a49bb8cb623e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:54:23.747942 11710 ts_tablet_manager.cc:1918] T 4b07a941bab2418dbc898429feef2339 P 2832c55a6ad84caab75d0152e0de0cbc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:54:23.748193 11710 log.cc:1196] T 4b07a941bab2418dbc898429feef2339 P 2832c55a6ad84caab75d0152e0de0cbc: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ts-2/wal/wals/4b07a941bab2418dbc898429feef2339
I20260806 02:54:23.748823 11710 ts_tablet_manager.cc:1939] T 4b07a941bab2418dbc898429feef2339 P 2832c55a6ad84caab75d0152e0de0cbc: Deleting consensus metadata
I20260806 02:54:23.749378 11711 ts_tablet_manager.cc:1905] T 4b07a941bab2418dbc898429feef2339 P 63328cf8c6e4496d9f39a49bb8cb623e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:54:23.749413 11709 ts_tablet_manager.cc:1905] T 4b07a941bab2418dbc898429feef2339 P 4bebec71961e45e7b7570d426e0cd61c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:54:23.751058 10925 catalog_manager.cc:4841] TS 2832c55a6ad84caab75d0152e0de0cbc (127.25.211.131:34253): tablet 4b07a941bab2418dbc898429feef2339 (table test-table [id=19116a9e23104f44a0c97d3cd35e9df4]) successfully deleted
Aug 06 02:54:23 dist-test-slave-zj8q krb5kdc[10232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.129@KRBTEST.COM
I20260806 02:54:23.757915 11709 ts_tablet_manager.cc:1918] T 4b07a941bab2418dbc898429feef2339 P 4bebec71961e45e7b7570d426e0cd61c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:54:23.758468 11709 log.cc:1196] T 4b07a941bab2418dbc898429feef2339 P 4bebec71961e45e7b7570d426e0cd61c: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ts-0/wal/wals/4b07a941bab2418dbc898429feef2339
I20260806 02:54:23.759176 11709 ts_tablet_manager.cc:1939] T 4b07a941bab2418dbc898429feef2339 P 4bebec71961e45e7b7570d426e0cd61c: Deleting consensus metadata
I20260806 02:54:23.761479 10923 catalog_manager.cc:4841] TS 4bebec71961e45e7b7570d426e0cd61c (127.25.211.129:43433): tablet 4b07a941bab2418dbc898429feef2339 (table test-table [id=19116a9e23104f44a0c97d3cd35e9df4]) successfully deleted
I20260806 02:54:23.761695 11711 ts_tablet_manager.cc:1918] T 4b07a941bab2418dbc898429feef2339 P 63328cf8c6e4496d9f39a49bb8cb623e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:54:23.761926 11711 log.cc:1196] T 4b07a941bab2418dbc898429feef2339 P 63328cf8c6e4496d9f39a49bb8cb623e: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1785984707115702-26446-0/minicluster-data/ts-1/wal/wals/4b07a941bab2418dbc898429feef2339
I20260806 02:54:23.762625 11711 ts_tablet_manager.cc:1939] T 4b07a941bab2418dbc898429feef2339 P 63328cf8c6e4496d9f39a49bb8cb623e: Deleting consensus metadata
I20260806 02:54:23.763929 11671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:23.727508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52452 (local address 127.25.211.129:43433)
0806 02:54:23.727718 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:23.727725 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:23.733739 (+ 6014us) server_negotiation.cc:407] Connection header received
0806 02:54:23.734876 (+ 1137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:23.734884 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:23.734980 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:23.735059 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:54:23.739628 (+ 4569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:23.740979 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:23.747310 (+ 6331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:23.747960 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:23.748105 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:23.755387 (+ 7282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:54:23.755409 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:54:23.755431 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:54:23.755483 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:54:23.760080 (+ 4597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:23.761322 (+ 1242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:23.761332 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:23.761336 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:23.761506 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:23.762052 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:23.762056 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:23.762059 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:23.762486 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:54:23.762724 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:23.763290 (+ 566us) server_negotiation.cc:299] Negotiation successful
0806 02:54:23.763521 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 02:54:23.764712 10925 catalog_manager.cc:4841] TS 63328cf8c6e4496d9f39a49bb8cb623e (127.25.211.130:35013): tablet 4b07a941bab2418dbc898429feef2339 (table test-table [id=19116a9e23104f44a0c97d3cd35e9df4]) successfully deleted
I20260806 02:54:23.764972 11252 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:54:23.859283 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 11061
I20260806 02:54:23.889426 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 11267
Aug 06 02:54:23 dist-test-slave-zj8q krb5kdc[10232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.130@KRBTEST.COM
I20260806 02:54:23.906314 11677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:23.890208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50686 (local address 127.25.211.130:35013)
0806 02:54:23.890401 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:23.890407 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:23.890483 (+ 76us) server_negotiation.cc:407] Connection header received
0806 02:54:23.890716 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:23.890721 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:23.890809 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:23.890912 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:54:23.892162 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:23.893008 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:23.894490 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:23.895015 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:23.895152 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:23.899443 (+ 4291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:54:23.899459 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:54:23.899482 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:54:23.899517 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:54:23.902873 (+ 3356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:23.903664 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:23.903672 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:23.903674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:23.903929 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:23.904421 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:23.904425 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:23.904428 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:23.904974 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:54:23.905301 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:23.905645 (+ 344us) server_negotiation.cc:299] Negotiation successful
0806 02:54:23.905895 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260806 02:54:23.907156 11448 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:54:24.024717 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 11267
I20260806 02:54:24.049445 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 11461
Aug 06 02:54:24 dist-test-slave-zj8q krb5kdc[10232](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.131@KRBTEST.COM
I20260806 02:54:24.064697 11673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:54:24.050270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37778 (local address 127.25.211.131:34253)
0806 02:54:24.050432 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0806 02:54:24.050436 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:54:24.050456 (+ 20us) server_negotiation.cc:407] Connection header received
0806 02:54:24.050622 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:54:24.050627 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:54:24.050691 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:54:24.050779 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:54:24.052117 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:24.052863 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:24.054294 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:54:24.054710 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:54:24.054814 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:54:24.059228 (+ 4414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:54:24.059244 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:54:24.059262 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:54:24.059310 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:54:24.062142 (+ 2832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:24.062867 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:24.062875 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:24.062877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:24.063016 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:54:24.063358 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:54:24.063361 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:54:24.063363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:54:24.063739 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:54:24.063948 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0806 02:54:24.064249 (+ 301us) server_negotiation.cc:299] Negotiation successful
0806 02:54:24.064418 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 02:54:24.065321 11637 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:54:24.155687 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 11461
I20260806 02:54:24.178668 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 10888
I20260806 02:54:24.179803 10989 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:54:24.294822 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 10888
I20260806 02:54:24.327158 26446 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260806 02:54:24.673411 26446 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260806 02:54:24.673760 26446 mini_ranger.cc:66] Stopping Ranger...
I20260806 02:54:25.024665 26446 mini_ranger.cc:68] Stopped Ranger
2026-08-06 02:54:25.024 UTC [10277] LOG: received smart shutdown request
2026-08-06 02:54:25.028 UTC [10277] LOG: background worker "logical replication launcher" (PID 10294) exited with exit code 1
2026-08-06 02:54:25.029 UTC [10284] LOG: shutting down
2026-08-06 02:54:25.213 UTC [10277] LOG: database system is shut down
2026-08-06T02:54:25Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (53153 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:54:25 dist-test-slave-zj8q krb5kdc[11740](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:54:25 dist-test-slave-zj8q krb5kdc[11740](info): set up 2 sockets
Aug 06 02:54:25 dist-test-slave-zj8q krb5kdc[11740](info): commencing operation
krb5kdc: starting...
W20260806 02:54:26.567601 26446 mini_kdc.cc:121] Time spent starting KDC: real 1.197s user 0.010s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Aug 06 02:54:26 dist-test-slave-zj8q krb5kdc[11740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:54:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:54:26Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.25.211.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.25.211.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.25.211.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/krb5kdc/rangeradmin_127.25.211.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.25.211.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/krb5kdc/rangeradmin_127.25.211.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.25.211.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.25.211.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.25.211.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/krb5kdc/rangerlookup_127.25.211.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.25.211.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/krb5kdc/rangerlookup_127.25.211.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.25.211.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.25.211.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.211.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/krb5kdc/HTTP_127.25.211.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.25.211.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/krb5kdc/HTTP_127.25.211.148@KRBTEST.COM.keytab.
I20260806 02:54:26.992740 26446 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskFsNc4K/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/postgres -l logfile start
2026-08-06 02:54:29.497 UTC [11782] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-06 02:54:29.497 UTC [11782] LOG: listening on IPv4 address "127.25.211.148", port 52091
2026-08-06 02:54:29.500 UTC [11782] LOG: listening on Unix socket "/tmp/.s.PGSQL.52091"
2026-08-06 02:54:29.521 UTC [11786] LOG: database system was shut down at 2026-08-06 02:54:28 UTC
2026-08-06 02:54:29.524 UTC [11782] LOG: database system is ready to accept connections
I20260806 02:54:30.620040 26446 mini_postgres.cc:96] Postgres bound to 52091
2026-08-06 02:54:30.647 UTC [11799] FATAL: database "slave" does not exist
127.25.211.148:52091 - accepting connections
I20260806 02:54:30.647933 26446 mini_ranger.cc:161] Starting Ranger...
I20260806 02:54:30.681830 26446 mini_ranger.cc:84] Created miniranger Postgres user
I20260806 02:54:31.179876 26446 mini_ranger.cc:87] Created ranger Postgres database
I20260806 02:54:31.180006 26446 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ranger-admin
2026-08-06 02:54:31,487 [I] DB FLAVOR :POSTGRES
2026-08-06 02:54:31,487 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 02:54:31,488 [I] Checking connection..
2026-08-06 02:54:31,488 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFsNc4K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFsNc4K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.211.148:52091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-06 02:54:31,751 [I] Checking connection passed.
2026-08-06 02:54:31,751 [I] --------- Verifying version history table ---------
2026-08-06 02:54:31,752 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFsNc4K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFsNc4K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.211.148:52091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 02:54:32,020 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 02:54:32,021 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFsNc4K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFsNc4K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.211.148:52091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 02:54:32,297 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 02:54:32,298 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-06 02:54:32,298 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFsNc4K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFsNc4K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.211.148:52091/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskFsNc4K/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-06 02:54:32.564 UTC [11890] WARNING: there is no transaction in progress
2026-08-06 02:54:32,573 [I] create_dbversion_catalog.sql file imported successfully
2026-08-06 02:54:32,573 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFsNc4K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFsNc4K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.211.148:52091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 02:54:32,822 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-06 02:54:32,823 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 02:54:32,823 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFsNc4K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFsNc4K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.211.148:52091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-06 02:54:33,081 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFsNc4K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFsNc4K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.211.148:52091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-06 02:54:33,335 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFsNc4K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFsNc4K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.211.148:52091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-zj8q.c.gcp-upstream.internal','N') ;"
2026-08-06 02:54:33,589 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFsNc4K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFsNc4K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.211.148:52091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 02:54:33,841 [I] Table x_portal_user does not exist in database ranger
2026-08-06 02:54:33,841 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFsNc4K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFsNc4K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.211.148:52091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 02:54:34,109 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-06 02:54:34,109 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFsNc4K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFsNc4K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.211.148:52091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 02:54:34,358 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFsNc4K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFsNc4K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.211.148:52091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 02:54:34,625 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-06 02:54:34,626 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFsNc4K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFsNc4K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.211.148:52091/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskFsNc4K/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-06 02:54:35.693 UTC [12056] WARNING: there is no transaction in progress
2026-08-06 02:54:35.710 UTC [12056] WARNING: there is no transaction in progress
2026-08-06 02:54:35.733 UTC [12056] WARNING: there is no transaction in progress
2026-08-06 02:54:35.758 UTC [12056] WARNING: there is no transaction in progress
2026-08-06 02:54:35.790 UTC [12056] WARNING: there is no transaction in progress
2026-08-06 02:54:35.817 UTC [12056] WARNING: there is no transaction in progress
2026-08-06 02:54:35.906 UTC [12056] WARNING: there is no transaction in progress
2026-08-06 02:54:35.919 UTC [12056] WARNING: there is no transaction in progress
2026-08-06 02:54:35.944 UTC [12056] WARNING: there is no transaction in progress
2026-08-06 02:54:35.961 UTC [12056] WARNING: there is no transaction in progress
2026-08-06 02:54:35.987 UTC [12056] WARNING: there is no transaction in progress
2026-08-06 02:54:36.016 UTC [12056] WARNING: there is no transaction in progress
2026-08-06 02:54:36.036 UTC [12056] WARNING: there is no transaction in progress
2026-08-06 02:54:36.049 UTC [12056] WARNING: there is no transaction in progress
2026-08-06 02:54:36.061 UTC [12056] WARNING: there is no transaction in progress
2026-08-06 02:54:36.074 UTC [12056] WARNING: there is no transaction in progress
2026-08-06 02:54:37,018 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-06 02:54:37,019 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFsNc4K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFsNc4K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.211.148:52091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-06 02:54:37,486 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-06 02:54:37,487 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFsNc4K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFsNc4K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.211.148:52091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 02:54:38,012 [I] Table x_portal_user already exists in database 'ranger'
2026-08-06 02:54:38,013 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFsNc4K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFsNc4K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.211.148:52091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 02:54:38,507 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-06 02:54:38,507 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFsNc4K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFsNc4K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.211.148:52091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 02:54:38,776 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFsNc4K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFsNc4K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.211.148:52091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 02:54:39,026 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFsNc4K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFsNc4K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.211.148:52091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-zj8q.c.gcp-upstream.internal';"
2026-08-06 02:54:39,282 [I] CORE_DB_SCHEMA import status has been updated
2026-08-06 02:54:39,282 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFsNc4K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFsNc4K/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.211.148:52091/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-06 02:54:39,554 [I] DB_PATCHES have already been applied
I20260806 02:54:39.560360 26446 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ranger-admin:/tmp/dist-test-taskFsNc4K/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskFsNc4K/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskFsNc4K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFsNc4K/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260806 02:54:39.560462 26446 mini_ranger.cc:193] Using host: 127.25.211.148
I20260806 02:54:39.582109 26446 mini_ranger.cc:227] Ranger admin URL: http://127.25.211.148:37967
Aug 06, 2026 2:54:39 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 2:54:39 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 2:54:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ranger-admin/ews/webapp
Aug 06, 2026 2:54:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 06, 2026 2:54:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 06, 2026 2:54:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ranger-admin/ews/webapp].
Aug 06, 2026 2:54:40 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260806 02:54:40.844642 26446 mini_ranger.cc:160] Time spent starting Ranger: real 10.197s user 0.000s sys 0.068s
Aug 06, 2026 2:55:01 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 06, 2026 2:55:02 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 06, 2026 2:55:02 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 06, 2026 2:55:02 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 06, 2026 2:55:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 06, 2026 2:55:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 06, 2026 2:55:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 06, 2026 2:55:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 06, 2026 2:55:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 06, 2026 2:55:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 06, 2026 2:55:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 06, 2026 2:55:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 06, 2026 2:55:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 06, 2026 2:55:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 06, 2026 2:55:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 06, 2026 2:55:02 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 06, 2026 2:55:02 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 06, 2026 2:55:03 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260806 02:55:03.652877 26446 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.25.211.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.25.211.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.25.211.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/krb5kdc/rangerkms_127.25.211.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.25.211.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/krb5kdc/rangerkms_127.25.211.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.25.211.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.25.211.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.25.211.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/krb5kdc/HTTP_127.25.211.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.25.211.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/krb5kdc/HTTP_127.25.211.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 06 02:55:03 dist-test-slave-zj8q krb5kdc[11740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984903, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260806 02:55:03.860689 26446 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260806 02:55:03.889942 26446 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260806 02:55:04.763820 26446 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260806 02:55:04.763939 26446 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ranger-kms
I20260806 02:55:04.763983 26446 mini_ranger_kms.cc:227] Using postgres at 127.25.211.148:52091
2026-08-06 02:55:05,139 [I] DB FLAVOR :POSTGRES
2026-08-06 02:55:05,139 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 02:55:05,140 [I] Checking connection
2026-08-06 02:55:05,140 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFsNc4K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.211.148:52091/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-06 02:55:05,417 [I] connection success
2026-08-06 02:55:05,417 [I] --------- Verifying Ranger DB tables ---------
2026-08-06 02:55:05,417 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-06 02:55:05,418 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFsNc4K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.211.148:52091/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-06 02:55:05,688 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-06 02:55:05,688 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 02:55:05,688 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-06 02:55:05,688 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFsNc4K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.25.211.148:52091/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-06 02:55:05,987 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260806 02:55:06.503569 26446 mini_ranger_kms.cc:310] Created kms service
I20260806 02:55:06.679816 26446 mini_ranger_kms.cc:326] Created kudu user
I20260806 02:55:06.721968 26446 mini_ranger_kms.cc:343] Created rangerkms user
Aug 06 02:55:06 dist-test-slave-zj8q krb5kdc[11740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984903, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.25.211.148@KRBTEST.COM
I20260806 02:55:07.143446 26446 mini_ranger_kms.cc:384] Added ranger policy
I20260806 02:55:07.143781 26446 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ranger-kms:/tmp/dist-test-taskFsNc4K/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFsNc4K/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskFsNc4K/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260806 02:55:07.143826 26446 mini_ranger_kms.cc:240] Using host: 127.25.211.148
I20260806 02:55:07.171218 26446 mini_ranger_kms.cc:276] Ranger KMS PID: 12335
I20260806 02:55:07.171360 26446 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.25.211.148:60817
Aug 06, 2026 2:55:08 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 2:55:08 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 2:55:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 06, 2026 2:55:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 06, 2026 2:55:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ranger-kms/ews/webapp].
Aug 06, 2026 2:55:08 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 06 02:55:09 dist-test-slave-zj8q krb5kdc[11740](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785984909, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.25.211.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:55:09.642282 26446 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.782s user 0.076s sys 0.343s
I20260806 02:55:09.642532 26446 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260806 02:55:09.642783 26446 mini_ranger_kms.cc:401] 127.25.211.148:60817/kms/v1/keys
Aug 06 02:55:09 dist-test-slave-zj8q krb5kdc[11740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984909, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.25.211.148@KRBTEST.COM for HTTP/127.25.211.148@KRBTEST.COM
Aug 06 02:55:10 dist-test-slave-zj8q krb5kdc[11740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984909, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.25.211.148@KRBTEST.COM for HTTP/127.25.211.148@KRBTEST.COM
Aug 06 02:55:12 dist-test-slave-zj8q krb5kdc[11740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.25.211.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab.
Entry for principal kudu/127.25.211.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab.
I20260806 02:55:12.722391 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:42641
--webserver_interface=127.25.211.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:37055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.211.190:42641
--ranger_config_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.211.148:60817/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:55:13.079391 12412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:55:13.079715 12412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:55:13.079849 12412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:55:13.087384 12412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:55:13.087513 12412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:55:13.087563 12412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:55:13.087648 12412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:55:13.087713 12412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:55:13.098286 12412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:37055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/master-0/wal
--ranger_kms_url=127.25.211.148:60817/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.25.211.190:42641
--ranger_config_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:42641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.211.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:55:13.099692 12412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:55:13.101235 12412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:55:13.114526 12421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:13.114457 12425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:13.114492 12423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:55:13.114915 12412 server_base.cc:1034] running on GCE node
I20260806 02:55:13.115873 12412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:55:13.117383 12412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:55:13.118593 12412 hybrid_clock.cc:648] HybridClock initialized: now 1785984913118530 us; error 51 us; skew 500 ppm
I20260806 02:55:13.118974 12412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:55:13 dist-test-slave-zj8q krb5kdc[11740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984913, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:55:13.127135 12412 init.cc:377] Logged in from keytab as kudu/127.25.211.190@KRBTEST.COM (short username kudu)
I20260806 02:55:13.128839 12412 webserver.cc:460] Webserver started at http://127.25.211.190:36357/ using document root <none> and password file <none>
I20260806 02:55:13.129562 12412 fs_manager.cc:359] Metadata directory not provided
I20260806 02:55:13.129691 12412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:55:13.130023 12412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 02:55:13 dist-test-slave-zj8q krb5kdc[11740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984913, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.190@KRBTEST.COM for HTTP/127.25.211.148@KRBTEST.COM
I20260806 02:55:13.183342 12412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/master-0/data/instance:
uuid: "107613bd63d14cd1a485cbab9a3cb60a"
format_stamp: "Formatted at 2026-08-06 02:55:13 on dist-test-slave-zj8q"
server_key: "5b73e0533b55c5a20bf8a92543a46f26"
server_key_iv: "28778025e94f552c2372128e5af4f6af"
server_key_version: "kuduclusterkey@0"
I20260806 02:55:13.184387 12412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/master-0/wal/instance:
uuid: "107613bd63d14cd1a485cbab9a3cb60a"
format_stamp: "Formatted at 2026-08-06 02:55:13 on dist-test-slave-zj8q"
server_key: "5b73e0533b55c5a20bf8a92543a46f26"
server_key_iv: "28778025e94f552c2372128e5af4f6af"
server_key_version: "kuduclusterkey@0"
I20260806 02:55:13.190444 12412 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260806 02:55:13.208341 12435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:13.209851 12412 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:55:13.210071 12412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/master-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/master-0/wal
uuid: "107613bd63d14cd1a485cbab9a3cb60a"
format_stamp: "Formatted at 2026-08-06 02:55:13 on dist-test-slave-zj8q"
server_key: "5b73e0533b55c5a20bf8a92543a46f26"
server_key_iv: "28778025e94f552c2372128e5af4f6af"
server_key_version: "kuduclusterkey@0"
I20260806 02:55:13.210386 12412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:55:13.259421 12412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:55:13.263722 12412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:55:13.264048 12412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:55:13.283094 12412 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.190:42641
I20260806 02:55:13.283147 12509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.190:42641 every 8 connection(s)
I20260806 02:55:13.284579 12412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
I20260806 02:55:13.287642 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 12412
I20260806 02:55:13.287890 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/master-0/wal/instance
I20260806 02:55:13.288857 26446 external_mini_cluster.cc:1428] Setting key 7159ca79117fef8821d2830f698e450c
I20260806 02:55:13.289130 12510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:55:13.304152 12510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 107613bd63d14cd1a485cbab9a3cb60a: Bootstrap starting.
Aug 06 02:55:13 dist-test-slave-zj8q krb5kdc[11740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:55:13.307813 12510 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 107613bd63d14cd1a485cbab9a3cb60a: Neither blocks nor log segments found. Creating new log.
I20260806 02:55:13.309130 12510 log.cc:824] T 00000000000000000000000000000000 P 107613bd63d14cd1a485cbab9a3cb60a: Log is configured to *not* fsync() on all Append() calls
I20260806 02:55:13.312361 12510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 107613bd63d14cd1a485cbab9a3cb60a: No bootstrap required, opened a new log
I20260806 02:55:13.313151 12513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:13.291052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53076 (local address 127.25.211.190:42641)
0806 02:55:13.291893 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:13.291905 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:13.291951 (+ 46us) server_negotiation.cc:407] Connection header received
0806 02:55:13.293928 (+ 1977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:13.293951 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:13.294410 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:13.294826 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:13.296580 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:13.297963 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:13.299898 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:13.300680 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:13.300855 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:13.305604 (+ 4749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:13.305614 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:13.305633 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:13.305681 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:13.308909 (+ 3228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:13.309676 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:13.309683 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:13.309689 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:13.309915 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:13.310405 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:13.310409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:13.310411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:13.310929 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:13.311213 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:13.311684 (+ 471us) server_negotiation.cc:299] Negotiation successful
0806 02:55:13.312082 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":395,"threads_started":1}
I20260806 02:55:13.320283 12510 raft_consensus.cc:348] T 00000000000000000000000000000000 P 107613bd63d14cd1a485cbab9a3cb60a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "107613bd63d14cd1a485cbab9a3cb60a" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 42641 } }
I20260806 02:55:13.320695 12510 raft_consensus.cc:374] T 00000000000000000000000000000000 P 107613bd63d14cd1a485cbab9a3cb60a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:55:13.320793 12510 raft_consensus.cc:729] T 00000000000000000000000000000000 P 107613bd63d14cd1a485cbab9a3cb60a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 107613bd63d14cd1a485cbab9a3cb60a, State: Initialized, Role: FOLLOWER
I20260806 02:55:13.321395 12510 consensus_queue.cc:260] T 00000000000000000000000000000000 P 107613bd63d14cd1a485cbab9a3cb60a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "107613bd63d14cd1a485cbab9a3cb60a" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 42641 } }
I20260806 02:55:13.321628 12510 raft_consensus.cc:388] T 00000000000000000000000000000000 P 107613bd63d14cd1a485cbab9a3cb60a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:55:13.321839 12510 raft_consensus.cc:482] T 00000000000000000000000000000000 P 107613bd63d14cd1a485cbab9a3cb60a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:55:13.322021 12510 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 107613bd63d14cd1a485cbab9a3cb60a [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:55:13.324851 12510 raft_consensus.cc:504] T 00000000000000000000000000000000 P 107613bd63d14cd1a485cbab9a3cb60a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "107613bd63d14cd1a485cbab9a3cb60a" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 42641 } }
I20260806 02:55:13.325345 12510 leader_election.cc:302] T 00000000000000000000000000000000 P 107613bd63d14cd1a485cbab9a3cb60a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 107613bd63d14cd1a485cbab9a3cb60a; no voters:
I20260806 02:55:13.325927 12510 leader_election.cc:288] T 00000000000000000000000000000000 P 107613bd63d14cd1a485cbab9a3cb60a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:55:13.326030 12517 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 107613bd63d14cd1a485cbab9a3cb60a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:55:13.326867 12517 raft_consensus.cc:686] T 00000000000000000000000000000000 P 107613bd63d14cd1a485cbab9a3cb60a [term 1 LEADER]: Becoming Leader. State: Replica: 107613bd63d14cd1a485cbab9a3cb60a, State: Running, Role: LEADER
I20260806 02:55:13.327347 12517 consensus_queue.cc:237] T 00000000000000000000000000000000 P 107613bd63d14cd1a485cbab9a3cb60a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "107613bd63d14cd1a485cbab9a3cb60a" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 42641 } }
I20260806 02:55:13.328008 12510 sys_catalog.cc:564] T 00000000000000000000000000000000 P 107613bd63d14cd1a485cbab9a3cb60a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:55:13.329968 12520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 107613bd63d14cd1a485cbab9a3cb60a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 107613bd63d14cd1a485cbab9a3cb60a. Latest consensus state: current_term: 1 leader_uuid: "107613bd63d14cd1a485cbab9a3cb60a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "107613bd63d14cd1a485cbab9a3cb60a" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 42641 } } }
I20260806 02:55:13.330422 12520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 107613bd63d14cd1a485cbab9a3cb60a [sys.catalog]: This master's current role is: LEADER
I20260806 02:55:13.331161 12519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 107613bd63d14cd1a485cbab9a3cb60a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "107613bd63d14cd1a485cbab9a3cb60a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "107613bd63d14cd1a485cbab9a3cb60a" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 42641 } } }
I20260806 02:55:13.331459 12519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 107613bd63d14cd1a485cbab9a3cb60a [sys.catalog]: This master's current role is: LEADER
I20260806 02:55:13.338132 12526 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:55:13.338439 12510 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 02:55:13.353236 12526 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:55:13.366437 12526 catalog_manager.cc:1347] Generated new cluster ID: 16010e50a31b49eca7237ace11dc2f5e
I20260806 02:55:13.366568 12526 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:55:13.393325 12526 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:55:13.395534 12526 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:55:13.411101 12526 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 107613bd63d14cd1a485cbab9a3cb60a: Generated new TSK 0
I20260806 02:55:13.412341 12526 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.211.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab.
Entry for principal kudu/127.25.211.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab.
I20260806 02:55:14.126554 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.211.148:60817/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.129:0
--local_ip_for_outbound_sockets=127.25.211.129
--webserver_interface=127.25.211.129
--webserver_port=0
--tserver_master_addrs=127.25.211.190:42641
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab
--principal=kudu/127.25.211.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:37055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:55:14 dist-test-slave-zj8q krb5kdc[11740](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785984914, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260806 02:55:14.597895 12574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:55:14.598413 12574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:55:14.598575 12574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:55:14.608806 12574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:55:14.608964 12574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:55:14.609127 12574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.129
I20260806 02:55:14.621181 12574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:37055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.25.211.148:60817/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab
--principal=kudu/127.25.211.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.211.129
--webserver_port=0
--tserver_master_addrs=127.25.211.190:42641
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.129
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:55:14.622745 12574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:55:14.624150 12574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:55:14.644650 12586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:14.645205 12587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:55:14.650882 12574 server_base.cc:1034] running on GCE node
W20260806 02:55:14.651217 12591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:55:14.652076 12574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:55:14.653200 12574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:55:14.654450 12574 hybrid_clock.cc:648] HybridClock initialized: now 1785984914654390 us; error 52 us; skew 500 ppm
I20260806 02:55:14.654807 12574 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:55:14 dist-test-slave-zj8q krb5kdc[11740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984914, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:55:14.658329 12574 init.cc:377] Logged in from keytab as kudu/127.25.211.129@KRBTEST.COM (short username kudu)
I20260806 02:55:14.665381 12574 webserver.cc:460] Webserver started at http://127.25.211.129:41395/ using document root <none> and password file <none>
I20260806 02:55:14.666296 12574 fs_manager.cc:359] Metadata directory not provided
I20260806 02:55:14.666440 12574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:55:14.666786 12574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 02:55:14 dist-test-slave-zj8q krb5kdc[11740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984914, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.129@KRBTEST.COM for HTTP/127.25.211.148@KRBTEST.COM
I20260806 02:55:14.686183 12574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-0/data/instance:
uuid: "372649a493804ce487faf5268d7fb947"
format_stamp: "Formatted at 2026-08-06 02:55:14 on dist-test-slave-zj8q"
server_key: "b8f7bd24d41fb3cbec0a7b11407028c8"
server_key_iv: "26e0e033172eb1e50fa6a6cca683d70c"
server_key_version: "kuduclusterkey@0"
I20260806 02:55:14.689622 12574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-0/wal/instance:
uuid: "372649a493804ce487faf5268d7fb947"
format_stamp: "Formatted at 2026-08-06 02:55:14 on dist-test-slave-zj8q"
server_key: "b8f7bd24d41fb3cbec0a7b11407028c8"
server_key_iv: "26e0e033172eb1e50fa6a6cca683d70c"
server_key_version: "kuduclusterkey@0"
I20260806 02:55:14.702196 12574 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.003s sys 0.006s
I20260806 02:55:14.729409 12606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:14.736099 12574 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.001s sys 0.001s
I20260806 02:55:14.736354 12574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-0/wal
uuid: "372649a493804ce487faf5268d7fb947"
format_stamp: "Formatted at 2026-08-06 02:55:14 on dist-test-slave-zj8q"
server_key: "b8f7bd24d41fb3cbec0a7b11407028c8"
server_key_iv: "26e0e033172eb1e50fa6a6cca683d70c"
server_key_version: "kuduclusterkey@0"
I20260806 02:55:14.736608 12574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:55:14.752698 12574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:55:14.757315 12574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:55:14.757637 12574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:55:14.758829 12574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:55:14.760525 12574 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:55:14.760615 12574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:14.760778 12574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:55:14.760941 12574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:14.849629 12574 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.129:43113
I20260806 02:55:14.849653 12763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.129:43113 every 8 connection(s)
I20260806 02:55:14.851188 12574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
I20260806 02:55:14.853570 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 12574
I20260806 02:55:14.853777 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-0/wal/instance
I20260806 02:55:14.854625 26446 external_mini_cluster.cc:1428] Setting key 92dd970efe3599e1c620513b6a5a02e2
Aug 06 02:55:14 dist-test-slave-zj8q krb5kdc[11740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984914, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.129@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.211.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.130@KRBTEST.COM" created.
I20260806 02:55:14.915958 12767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:14.854668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:44453 (local address 127.25.211.190:42641)
0806 02:55:14.888479 (+ 33811us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:14.888485 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:14.888509 (+ 24us) server_negotiation.cc:407] Connection header received
0806 02:55:14.888578 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:14.888583 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:14.888689 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:14.888817 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:14.891523 (+ 2706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:14.892684 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:14.895274 (+ 2590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:14.895910 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:14.896014 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:14.903289 (+ 7275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:14.903299 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:14.903304 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:14.903359 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:14.911284 (+ 7925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:14.912516 (+ 1232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:14.912523 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:14.912526 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:14.912712 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:14.913366 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:14.913371 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:14.913375 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:14.913850 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:14.914001 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:14.915295 (+ 1294us) server_negotiation.cc:299] Negotiation successful
0806 02:55:14.915565 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33668,"thread_start_us":33515,"threads_started":1}
I20260806 02:55:14.918705 12770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:14.855784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:42641 (local address 127.25.211.129:44453)
0806 02:55:14.884809 (+ 29025us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:14.884872 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0806 02:55:14.887610 (+ 2738us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:55:14.889377 (+ 1767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:55:14.889391 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:55:14.890117 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:55:14.891173 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:14.891270 (+ 97us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:14.892922 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:14.892928 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:14.895073 (+ 2145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:14.895091 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:14.896389 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:14.896396 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:14.896698 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:14.897804 (+ 1106us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:55:14.897835 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:55:14.902892 (+ 5057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:55:14.911571 (+ 8679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:14.911584 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:14.911606 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:14.912332 (+ 726us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:14.912910 (+ 578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:14.912915 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:14.912919 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:14.913212 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:14.914104 (+ 892us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:55:14.914112 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:55:14.914664 (+ 552us) client_negotiation.cc:769] Sending connection context
0806 02:55:14.916538 (+ 1874us) client_negotiation.cc:240] Negotiation successful
0806 02:55:14.917163 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28690,"thread_start_us":28443,"threads_started":1}
I20260806 02:55:14.921155 12764 heartbeater.cc:347] Connected to a master server at 127.25.211.190:42641
I20260806 02:55:14.921527 12764 heartbeater.cc:464] Registering TS with master...
I20260806 02:55:14.922479 12764 heartbeater.cc:511] Master 127.25.211.190:42641 requested a full tablet report, sending...
I20260806 02:55:14.925092 12459 ts_manager.cc:194] Registered new tserver with Master: 372649a493804ce487faf5268d7fb947 (127.25.211.129:43113)
I20260806 02:55:14.927142 12459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.211.129@KRBTEST.COM'} at 127.25.211.129:44453
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab.
Entry for principal kudu/127.25.211.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab.
W20260806 02:55:14.958449 26446 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.25.211.130: real 0.101s user 0.002s sys 0.025s
I20260806 02:55:14.959385 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.211.148:60817/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.130:0
--local_ip_for_outbound_sockets=127.25.211.130
--webserver_interface=127.25.211.130
--webserver_port=0
--tserver_master_addrs=127.25.211.190:42641
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab
--principal=kudu/127.25.211.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:37055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:55:14 dist-test-slave-zj8q krb5kdc[11740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984914, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.190@KRBTEST.COM for HTTP/127.25.211.148@KRBTEST.COM
Aug 06 02:55:15 dist-test-slave-zj8q krb5kdc[11740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984914, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.190@KRBTEST.COM for HTTP/127.25.211.148@KRBTEST.COM
W20260806 02:55:15.518640 12774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:55:15.519181 12774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:55:15.519397 12774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:55:15.531270 12774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:55:15.531528 12774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:55:15.531776 12774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.130
I20260806 02:55:15.547830 12774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:37055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.25.211.148:60817/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab
--principal=kudu/127.25.211.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.211.130
--webserver_port=0
--tserver_master_addrs=127.25.211.190:42641
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.130
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:55:15.550009 12774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:55:15.552006 12774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:55:15.592501 12781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:15.594519 12783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:15.594625 12789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:55:15.595412 12774 server_base.cc:1034] running on GCE node
I20260806 02:55:15.596426 12774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:55:15.603142 12774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:55:15.604862 12774 hybrid_clock.cc:648] HybridClock initialized: now 1785984915604744 us; error 121 us; skew 500 ppm
I20260806 02:55:15.605422 12774 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:55:15 dist-test-slave-zj8q krb5kdc[11740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984915, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:55:15.610224 12774 init.cc:377] Logged in from keytab as kudu/127.25.211.130@KRBTEST.COM (short username kudu)
I20260806 02:55:15.631328 12774 webserver.cc:460] Webserver started at http://127.25.211.130:37251/ using document root <none> and password file <none>
I20260806 02:55:15.633862 12774 fs_manager.cc:359] Metadata directory not provided
I20260806 02:55:15.634142 12774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:55:15.634690 12774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 02:55:15 dist-test-slave-zj8q krb5kdc[11740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984915, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.130@KRBTEST.COM for HTTP/127.25.211.148@KRBTEST.COM
I20260806 02:55:15.679291 12774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-1/data/instance:
uuid: "c1bd0f8a5f954652b72b6b38401d795b"
format_stamp: "Formatted at 2026-08-06 02:55:15 on dist-test-slave-zj8q"
server_key: "0ba2e77ffcb5708a0052edee23b41e03"
server_key_iv: "0c184d987321dd13670969f74e98c79e"
server_key_version: "kuduclusterkey@0"
I20260806 02:55:15.699162 12774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-1/wal/instance:
uuid: "c1bd0f8a5f954652b72b6b38401d795b"
format_stamp: "Formatted at 2026-08-06 02:55:15 on dist-test-slave-zj8q"
server_key: "0ba2e77ffcb5708a0052edee23b41e03"
server_key_iv: "0c184d987321dd13670969f74e98c79e"
server_key_version: "kuduclusterkey@0"
I20260806 02:55:15.713802 12774 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.012s sys 0.000s
I20260806 02:55:15.751514 12807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:15.753688 12774 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.003s sys 0.007s
I20260806 02:55:15.753988 12774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-1/wal
uuid: "c1bd0f8a5f954652b72b6b38401d795b"
format_stamp: "Formatted at 2026-08-06 02:55:15 on dist-test-slave-zj8q"
server_key: "0ba2e77ffcb5708a0052edee23b41e03"
server_key_iv: "0c184d987321dd13670969f74e98c79e"
server_key_version: "kuduclusterkey@0"
I20260806 02:55:15.754396 12774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:55:15.805567 12774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:55:15.811040 12774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:55:15.811446 12774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:55:15.820575 12774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:55:15.828896 12774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:55:15.829109 12774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260806 02:55:15.829286 12774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:55:15.829382 12774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:15.931324 12764 heartbeater.cc:503] Master 127.25.211.190:42641 was elected leader, sending a full tablet report...
I20260806 02:55:16.187820 12774 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.130:35413
I20260806 02:55:16.187923 12968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.130:35413 every 8 connection(s)
I20260806 02:55:16.189683 12774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
I20260806 02:55:16.190719 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 12774
I20260806 02:55:16.190913 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-1/wal/instance
I20260806 02:55:16.191452 26446 external_mini_cluster.cc:1428] Setting key 2188cd55d69f5aa02a78c7c4099e3429
Aug 06 02:55:16 dist-test-slave-zj8q krb5kdc[11740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984915, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.130@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:55:16.233600 12972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:16.194897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:47649 (local address 127.25.211.190:42641)
0806 02:55:16.210801 (+ 15904us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:16.210810 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:16.212097 (+ 1287us) server_negotiation.cc:407] Connection header received
0806 02:55:16.215018 (+ 2921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:16.215024 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:16.215117 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:16.215185 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:16.217032 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:16.218223 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:16.220545 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:16.221234 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:16.221366 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:16.225823 (+ 4457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:16.225830 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:16.225834 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:16.225878 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:16.229392 (+ 3514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:16.230554 (+ 1162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:16.230562 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:16.230566 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:16.230739 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:16.231331 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:16.231337 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:16.231340 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:16.231769 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:16.231930 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:16.232910 (+ 980us) server_negotiation.cc:299] Negotiation successful
0806 02:55:16.233192 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15719,"thread_start_us":15545,"threads_started":1}
WARNING: no policy specified for kudu/127.25.211.131@KRBTEST.COM; defaulting to no policy
I20260806 02:55:16.234858 12971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:16.193524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:42641 (local address 127.25.211.130:47649)
0806 02:55:16.211556 (+ 18032us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:16.211628 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0806 02:55:16.214613 (+ 2985us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:55:16.215711 (+ 1098us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:55:16.215720 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:55:16.216214 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:55:16.216828 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:16.216847 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:16.218393 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:16.218399 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:16.220372 (+ 1973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:16.220398 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:16.221502 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:16.221509 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:16.221795 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:16.222614 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:55:16.222649 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:55:16.225502 (+ 2853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:55:16.229600 (+ 4098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:16.229612 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:16.229634 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:16.230377 (+ 743us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:16.230878 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:16.230883 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:16.230887 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:16.231178 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:16.231919 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:55:16.231928 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:55:16.232438 (+ 510us) client_negotiation.cc:769] Sending connection context
0806 02:55:16.232783 (+ 345us) client_negotiation.cc:240] Negotiation successful
0806 02:55:16.233372 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17624,"thread_start_us":17379,"threads_started":1}
I20260806 02:55:16.236652 12969 heartbeater.cc:347] Connected to a master server at 127.25.211.190:42641
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.131@KRBTEST.COM" created.
I20260806 02:55:16.237051 12969 heartbeater.cc:464] Registering TS with master...
I20260806 02:55:16.237852 12969 heartbeater.cc:511] Master 127.25.211.190:42641 requested a full tablet report, sending...
I20260806 02:55:16.239414 12459 ts_manager.cc:194] Registered new tserver with Master: c1bd0f8a5f954652b72b6b38401d795b (127.25.211.130:35413)
I20260806 02:55:16.240396 12459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.211.130@KRBTEST.COM'} at 127.25.211.130:47649
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab.
Entry for principal kudu/127.25.211.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab.
I20260806 02:55:16.274191 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.211.148:60817/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.131:0
--local_ip_for_outbound_sockets=127.25.211.131
--webserver_interface=127.25.211.131
--webserver_port=0
--tserver_master_addrs=127.25.211.190:42641
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab
--principal=kudu/127.25.211.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:37055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:55:16.626983 12977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:55:16.627334 12977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:55:16.627482 12977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:55:16.635073 12977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:55:16.635216 12977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:55:16.635381 12977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.131
I20260806 02:55:16.645797 12977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:37055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.25.211.148:60817/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab
--principal=kudu/127.25.211.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.211.131
--webserver_port=0
--tserver_master_addrs=127.25.211.190:42641
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.131
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:55:16.647222 12977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:55:16.648532 12977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:55:16.662135 12987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:16.662281 12985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:16.662258 12983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:55:16.663355 12977 server_base.cc:1034] running on GCE node
I20260806 02:55:16.664017 12977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:55:16.665014 12977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:55:16.666266 12977 hybrid_clock.cc:648] HybridClock initialized: now 1785984916666208 us; error 46 us; skew 500 ppm
I20260806 02:55:16.666615 12977 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:55:16 dist-test-slave-zj8q krb5kdc[11740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984916, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:55:16.669864 12977 init.cc:377] Logged in from keytab as kudu/127.25.211.131@KRBTEST.COM (short username kudu)
I20260806 02:55:16.671525 12977 webserver.cc:460] Webserver started at http://127.25.211.131:45677/ using document root <none> and password file <none>
I20260806 02:55:16.672263 12977 fs_manager.cc:359] Metadata directory not provided
I20260806 02:55:16.672386 12977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:55:16.672693 12977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 02:55:16 dist-test-slave-zj8q krb5kdc[11740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984916, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.131@KRBTEST.COM for HTTP/127.25.211.148@KRBTEST.COM
I20260806 02:55:16.691169 12977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-2/data/instance:
uuid: "f09305aa77aa4bec9462ba1ea4230291"
format_stamp: "Formatted at 2026-08-06 02:55:16 on dist-test-slave-zj8q"
server_key: "30e3e506f6c07fcdffb04cf6a1693a4a"
server_key_iv: "4df08539b89294be0ebb03ab9b0b47d1"
server_key_version: "kuduclusterkey@0"
I20260806 02:55:16.692333 12977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-2/wal/instance:
uuid: "f09305aa77aa4bec9462ba1ea4230291"
format_stamp: "Formatted at 2026-08-06 02:55:16 on dist-test-slave-zj8q"
server_key: "30e3e506f6c07fcdffb04cf6a1693a4a"
server_key_iv: "4df08539b89294be0ebb03ab9b0b47d1"
server_key_version: "kuduclusterkey@0"
I20260806 02:55:16.700246 12977 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260806 02:55:16.720038 12996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:16.721868 12977 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:55:16.722095 12977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-2/wal
uuid: "f09305aa77aa4bec9462ba1ea4230291"
format_stamp: "Formatted at 2026-08-06 02:55:16 on dist-test-slave-zj8q"
server_key: "30e3e506f6c07fcdffb04cf6a1693a4a"
server_key_iv: "4df08539b89294be0ebb03ab9b0b47d1"
server_key_version: "kuduclusterkey@0"
I20260806 02:55:16.722426 12977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:55:16.744719 12977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:55:16.749044 12977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:55:16.749337 12977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:55:16.750451 12977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:55:16.751911 12977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:55:16.751987 12977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:16.752084 12977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:55:16.752125 12977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:16.784047 12977 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.131:32939
I20260806 02:55:16.784088 13151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.131:32939 every 8 connection(s)
I20260806 02:55:16.785384 12977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
I20260806 02:55:16.794843 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 12977
I20260806 02:55:16.795022 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-2/wal/instance
I20260806 02:55:16.795502 26446 external_mini_cluster.cc:1428] Setting key 1ac9cf2cdcea55e7d59a66dc8b431060
Aug 06 02:55:16 dist-test-slave-zj8q krb5kdc[11740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984916, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.131@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:55:16.805575 13154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:16.787913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:37841 (local address 127.25.211.190:42641)
0806 02:55:16.788442 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:16.788449 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:16.789485 (+ 1036us) server_negotiation.cc:407] Connection header received
0806 02:55:16.791335 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:16.791340 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:16.791410 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:16.791508 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:16.793097 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:16.793851 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:16.795668 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:16.796175 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:16.796320 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:16.800099 (+ 3779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:16.800104 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:16.800108 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:16.800143 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:16.802646 (+ 2503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:16.803396 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:16.803400 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:16.803402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:16.803531 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:16.803958 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:16.803962 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:16.803964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:16.804259 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:16.804375 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:16.805110 (+ 735us) server_negotiation.cc:299] Negotiation successful
0806 02:55:16.805296 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":215,"threads_started":1}
I20260806 02:55:16.806672 13156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:16.788533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:42641 (local address 127.25.211.131:37841)
0806 02:55:16.789313 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:16.789353 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 02:55:16.791079 (+ 1726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:55:16.791783 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:55:16.791794 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:55:16.792242 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:55:16.792871 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:16.792890 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:16.794003 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:16.794007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:16.795498 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:16.795513 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:16.796457 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:16.796465 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:16.796780 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:16.797546 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:55:16.797578 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:55:16.799861 (+ 2283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:55:16.802804 (+ 2943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:16.802812 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:16.802827 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:16.803297 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:16.803610 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:16.803613 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:16.803615 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:16.803844 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:16.804356 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:55:16.804362 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:55:16.804751 (+ 389us) client_negotiation.cc:769] Sending connection context
0806 02:55:16.804983 (+ 232us) client_negotiation.cc:240] Negotiation successful
0806 02:55:16.805405 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":315,"threads_started":1}
I20260806 02:55:16.808337 13152 heartbeater.cc:347] Connected to a master server at 127.25.211.190:42641
I20260806 02:55:16.808707 13152 heartbeater.cc:464] Registering TS with master...
I20260806 02:55:16.809561 13152 heartbeater.cc:511] Master 127.25.211.190:42641 requested a full tablet report, sending...
I20260806 02:55:16.811383 12457 ts_manager.cc:194] Registered new tserver with Master: f09305aa77aa4bec9462ba1ea4230291 (127.25.211.131:32939)
I20260806 02:55:16.812397 12457 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.211.131@KRBTEST.COM'} at 127.25.211.131:37841
I20260806 02:55:16.824507 26446 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:55:16.826370 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 12574
Aug 06 02:55:16 dist-test-slave-zj8q krb5kdc[11740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.129@KRBTEST.COM
I20260806 02:55:16.845106 13159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:16.827754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47508 (local address 127.25.211.129:43113)
0806 02:55:16.828479 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:16.828487 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:16.828515 (+ 28us) server_negotiation.cc:407] Connection header received
0806 02:55:16.828599 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:16.828606 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:16.828848 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:16.829029 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:16.830580 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:16.831656 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:16.833401 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:16.834031 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:16.834139 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:16.838773 (+ 4634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:16.838791 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:16.838810 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:16.838853 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:16.842002 (+ 3149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:16.842843 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:16.842852 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:16.842854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:16.843017 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:16.843410 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:16.843414 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:16.843416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:16.843795 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:16.844115 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:16.844450 (+ 335us) server_negotiation.cc:299] Negotiation successful
0806 02:55:16.844750 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":354,"threads_started":1}
I20260806 02:55:16.846493 12758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:55:16.930493 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 12574
I20260806 02:55:16.955926 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 12774
Aug 06 02:55:16 dist-test-slave-zj8q krb5kdc[11740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.130@KRBTEST.COM
I20260806 02:55:16.977581 13163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:16.956791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38206 (local address 127.25.211.130:35413)
0806 02:55:16.957361 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:16.957368 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:16.957394 (+ 26us) server_negotiation.cc:407] Connection header received
0806 02:55:16.957461 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:16.957468 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:16.957679 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:16.957855 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:16.959223 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:16.960245 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:16.961714 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:16.962463 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:16.962564 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:16.971145 (+ 8581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:16.971164 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:16.971182 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:16.971229 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:16.974448 (+ 3219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:16.975364 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:16.975373 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:16.975375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:16.975543 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:16.975953 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:16.975956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:16.975958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:16.976355 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:16.976592 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:16.976932 (+ 340us) server_negotiation.cc:299] Negotiation successful
0806 02:55:16.977208 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":274,"threads_started":1}
I20260806 02:55:16.978984 12962 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:55:17.073765 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 12774
I20260806 02:55:17.097894 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 12977
Aug 06 02:55:17 dist-test-slave-zj8q krb5kdc[11740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.131@KRBTEST.COM
I20260806 02:55:17.115357 13166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:17.099021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43976 (local address 127.25.211.131:32939)
0806 02:55:17.099690 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:17.099698 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:17.099730 (+ 32us) server_negotiation.cc:407] Connection header received
0806 02:55:17.099790 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:17.099796 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:17.100014 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:17.100145 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:17.101453 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:17.102662 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:17.103960 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:17.104553 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:17.104671 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:17.109276 (+ 4605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:17.109292 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:17.109310 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:17.109362 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:17.112448 (+ 3086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:17.113225 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:17.113234 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:17.113236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:17.113374 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:17.113782 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:17.113785 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:17.113787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:17.114263 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:17.114517 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:17.114832 (+ 315us) server_negotiation.cc:299] Negotiation successful
0806 02:55:17.115041 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":309,"threads_started":1}
I20260806 02:55:17.116665 13147 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:55:17.200912 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 12977
I20260806 02:55:17.221963 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 12412
I20260806 02:55:17.223464 12502 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:55:17.332947 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 12412
I20260806 02:55:17.363276 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:42641
--webserver_interface=127.25.211.190
--webserver_port=36357
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:37055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.211.190:42641
--ranger_config_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.211.148:60817/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.25.211.148:60817/kms with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:55:17.707748 13169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:55:17.708067 13169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:55:17.708166 13169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:55:17.715787 13169 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:55:17.715924 13169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:55:17.715970 13169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:55:17.716027 13169 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:55:17.716075 13169 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:55:17.726645 13169 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:37055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.25.211.148:60817/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.25.211.190:42641
--ranger_config_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:42641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.211.190
--webserver_port=36357
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:55:17.727936 13169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:55:17.729264 13169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:55:17.741694 13175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:17.741806 13176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:17.741830 13178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:55:17.742923 13169 server_base.cc:1034] running on GCE node
I20260806 02:55:17.743865 13169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:55:17.745456 13169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:55:17.746693 13169 hybrid_clock.cc:648] HybridClock initialized: now 1785984917746640 us; error 50 us; skew 500 ppm
I20260806 02:55:17.747071 13169 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:55:17 dist-test-slave-zj8q krb5kdc[11740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984917, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:55:17.750371 13169 init.cc:377] Logged in from keytab as kudu/127.25.211.190@KRBTEST.COM (short username kudu)
I20260806 02:55:17.751986 13169 webserver.cc:460] Webserver started at http://127.25.211.190:36357/ using document root <none> and password file <none>
I20260806 02:55:17.752667 13169 fs_manager.cc:359] Metadata directory not provided
I20260806 02:55:17.752784 13169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 02:55:17 dist-test-slave-zj8q krb5kdc[11740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984917, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.190@KRBTEST.COM for HTTP/127.25.211.148@KRBTEST.COM
I20260806 02:55:17.807601 13169 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:55:17.810890 13188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:17.812417 13169 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:55:17.812598 13169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/master-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/master-0/wal
uuid: "107613bd63d14cd1a485cbab9a3cb60a"
format_stamp: "Formatted at 2026-08-06 02:55:13 on dist-test-slave-zj8q"
server_key: "5b73e0533b55c5a20bf8a92543a46f26"
server_key_iv: "28778025e94f552c2372128e5af4f6af"
server_key_version: "kuduclusterkey@0"
I20260806 02:55:17.813617 13169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:55:17.830993 13169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:55:17.835350 13169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:55:17.835611 13169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:55:17.854301 13169 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.190:42641
I20260806 02:55:17.854318 13263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.190:42641 every 8 connection(s)
I20260806 02:55:17.855896 13169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
I20260806 02:55:17.862831 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 13169
I20260806 02:55:17.863461 13265 sys_catalog.cc:263] Verifying existing consensus state
I20260806 02:55:17.863734 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.211.148:60817/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.129:43113
--local_ip_for_outbound_sockets=127.25.211.129
--tserver_master_addrs=127.25.211.190:42641
--webserver_port=41395
--webserver_interface=127.25.211.129
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab
--principal=kudu/127.25.211.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:37055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.25.211.148:60817/kms with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:55:17.866937 13265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 107613bd63d14cd1a485cbab9a3cb60a: Bootstrap starting.
I20260806 02:55:17.889398 13265 log.cc:824] T 00000000000000000000000000000000 P 107613bd63d14cd1a485cbab9a3cb60a: Log is configured to *not* fsync() on all Append() calls
I20260806 02:55:17.901849 13265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 107613bd63d14cd1a485cbab9a3cb60a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:55:17.902577 13265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 107613bd63d14cd1a485cbab9a3cb60a: Bootstrap complete.
I20260806 02:55:17.911741 13265 raft_consensus.cc:348] T 00000000000000000000000000000000 P 107613bd63d14cd1a485cbab9a3cb60a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "107613bd63d14cd1a485cbab9a3cb60a" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 42641 } }
I20260806 02:55:17.912674 13265 raft_consensus.cc:729] T 00000000000000000000000000000000 P 107613bd63d14cd1a485cbab9a3cb60a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 107613bd63d14cd1a485cbab9a3cb60a, State: Initialized, Role: FOLLOWER
I20260806 02:55:17.913336 13265 consensus_queue.cc:260] T 00000000000000000000000000000000 P 107613bd63d14cd1a485cbab9a3cb60a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "107613bd63d14cd1a485cbab9a3cb60a" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 42641 } }
I20260806 02:55:17.913563 13265 raft_consensus.cc:388] T 00000000000000000000000000000000 P 107613bd63d14cd1a485cbab9a3cb60a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:55:17.913681 13265 raft_consensus.cc:482] T 00000000000000000000000000000000 P 107613bd63d14cd1a485cbab9a3cb60a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:55:17.913853 13265 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 107613bd63d14cd1a485cbab9a3cb60a [term 1 FOLLOWER]: Advancing to term 2
I20260806 02:55:17.917855 13265 raft_consensus.cc:504] T 00000000000000000000000000000000 P 107613bd63d14cd1a485cbab9a3cb60a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "107613bd63d14cd1a485cbab9a3cb60a" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 42641 } }
I20260806 02:55:17.918442 13265 leader_election.cc:302] T 00000000000000000000000000000000 P 107613bd63d14cd1a485cbab9a3cb60a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 107613bd63d14cd1a485cbab9a3cb60a; no voters:
I20260806 02:55:17.919049 13265 leader_election.cc:288] T 00000000000000000000000000000000 P 107613bd63d14cd1a485cbab9a3cb60a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 02:55:17.919276 13274 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 107613bd63d14cd1a485cbab9a3cb60a [term 2 FOLLOWER]: Leader election won for term 2
I20260806 02:55:17.920830 13265 sys_catalog.cc:564] T 00000000000000000000000000000000 P 107613bd63d14cd1a485cbab9a3cb60a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:55:17.924340 13274 raft_consensus.cc:686] T 00000000000000000000000000000000 P 107613bd63d14cd1a485cbab9a3cb60a [term 2 LEADER]: Becoming Leader. State: Replica: 107613bd63d14cd1a485cbab9a3cb60a, State: Running, Role: LEADER
I20260806 02:55:17.925065 13274 consensus_queue.cc:237] T 00000000000000000000000000000000 P 107613bd63d14cd1a485cbab9a3cb60a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "107613bd63d14cd1a485cbab9a3cb60a" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 42641 } }
I20260806 02:55:17.927659 13279 sys_catalog.cc:455] T 00000000000000000000000000000000 P 107613bd63d14cd1a485cbab9a3cb60a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "107613bd63d14cd1a485cbab9a3cb60a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "107613bd63d14cd1a485cbab9a3cb60a" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 42641 } } }
I20260806 02:55:17.928054 13279 sys_catalog.cc:458] T 00000000000000000000000000000000 P 107613bd63d14cd1a485cbab9a3cb60a [sys.catalog]: This master's current role is: LEADER
I20260806 02:55:17.928597 13280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 107613bd63d14cd1a485cbab9a3cb60a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 107613bd63d14cd1a485cbab9a3cb60a. Latest consensus state: current_term: 2 leader_uuid: "107613bd63d14cd1a485cbab9a3cb60a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "107613bd63d14cd1a485cbab9a3cb60a" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 42641 } } }
I20260806 02:55:17.928853 13280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 107613bd63d14cd1a485cbab9a3cb60a [sys.catalog]: This master's current role is: LEADER
I20260806 02:55:17.952270 13291 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:55:17.960789 13291 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:55:17.963183 13291 catalog_manager.cc:1259] Loaded cluster ID: 16010e50a31b49eca7237ace11dc2f5e
I20260806 02:55:17.963384 13291 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:55:17.967505 13291 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:55:17.969563 13291 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 107613bd63d14cd1a485cbab9a3cb60a: Loaded TSK: 0
I20260806 02:55:17.971905 13291 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:55:17.974818 13265 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260806 02:55:18.355949 13268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:55:18.356424 13268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:55:18.356688 13268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:55:18.369112 13268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:55:18.369349 13268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:55:18.369577 13268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.129
I20260806 02:55:18.385820 13268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:37055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.25.211.148:60817/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab
--principal=kudu/127.25.211.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.129:43113
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.211.129
--webserver_port=41395
--tserver_master_addrs=127.25.211.190:42641
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.129
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:55:18.387945 13268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:55:18.389837 13268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:55:18.435387 13321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:18.441181 13320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:55:18.442538 13268 server_base.cc:1034] running on GCE node
W20260806 02:55:18.443177 13323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:55:18.444144 13268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:55:18.456432 13268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:55:18.459951 13268 hybrid_clock.cc:648] HybridClock initialized: now 1785984918459809 us; error 130 us; skew 500 ppm
I20260806 02:55:18.460511 13268 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:55:18 dist-test-slave-zj8q krb5kdc[11740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984918, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:55:18.469527 13268 init.cc:377] Logged in from keytab as kudu/127.25.211.129@KRBTEST.COM (short username kudu)
I20260806 02:55:18.471921 13268 webserver.cc:460] Webserver started at http://127.25.211.129:41395/ using document root <none> and password file <none>
I20260806 02:55:18.472991 13268 fs_manager.cc:359] Metadata directory not provided
I20260806 02:55:18.473280 13268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 02:55:18 dist-test-slave-zj8q krb5kdc[11740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984918, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.129@KRBTEST.COM for HTTP/127.25.211.148@KRBTEST.COM
I20260806 02:55:18.525833 13268 fs_manager.cc:709] Time spent opening directory manager: real 0.013s user 0.001s sys 0.008s
I20260806 02:55:18.530217 13336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:18.532398 13268 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 02:55:18.532609 13268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-0/wal
uuid: "372649a493804ce487faf5268d7fb947"
format_stamp: "Formatted at 2026-08-06 02:55:14 on dist-test-slave-zj8q"
server_key: "b8f7bd24d41fb3cbec0a7b11407028c8"
server_key_iv: "26e0e033172eb1e50fa6a6cca683d70c"
server_key_version: "kuduclusterkey@0"
I20260806 02:55:18.534096 13268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:55:18.572094 13268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:55:18.577771 13268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:55:18.578204 13268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:55:18.580929 13268 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:55:18.583312 13268 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:55:18.583503 13268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260806 02:55:18.583706 13268 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:55:18.583866 13268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:18.631211 13268 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.129:43113
I20260806 02:55:18.631315 13491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.129:43113 every 8 connection(s)
I20260806 02:55:18.633132 13268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
I20260806 02:55:18.635736 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 13268
I20260806 02:55:18.636624 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.211.148:60817/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.130:35413
--local_ip_for_outbound_sockets=127.25.211.130
--tserver_master_addrs=127.25.211.190:42641
--webserver_port=37251
--webserver_interface=127.25.211.130
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab
--principal=kudu/127.25.211.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:37055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.25.211.148:60817/kms with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:55:18 dist-test-slave-zj8q krb5kdc[11740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984918, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.129@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:55:18.703498 13498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:18.657754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:42641 (local address 127.25.211.129:48767)
0806 02:55:18.671580 (+ 13826us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:18.671661 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0806 02:55:18.677615 (+ 5954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:55:18.680311 (+ 2696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:55:18.680322 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:55:18.680941 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:55:18.681735 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:18.681764 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:18.684332 (+ 2568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:18.684339 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:18.686578 (+ 2239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:18.686609 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:18.688613 (+ 2004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:18.688621 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:18.688938 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:18.689735 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:55:18.689767 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:55:18.693234 (+ 3467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:55:18.697940 (+ 4706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:18.697952 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:18.697970 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:18.698719 (+ 749us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:18.699373 (+ 654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:18.699378 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:18.699382 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:18.699660 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:18.700713 (+ 1053us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:55:18.700731 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:55:18.701156 (+ 425us) client_negotiation.cc:769] Sending connection context
0806 02:55:18.701536 (+ 380us) client_negotiation.cc:240] Negotiation successful
0806 02:55:18.702190 (+ 654us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13404,"thread_start_us":14115,"threads_started":1}
I20260806 02:55:18.703508 13497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:18.657565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:48767 (local address 127.25.211.190:42641)
0806 02:55:18.672652 (+ 15087us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:18.672673 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:18.672736 (+ 63us) server_negotiation.cc:407] Connection header received
0806 02:55:18.678526 (+ 5790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:18.678550 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:18.679146 (+ 596us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:18.679565 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:18.682017 (+ 2452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:18.684069 (+ 2052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:18.686791 (+ 2722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:18.687920 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:18.688082 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:18.693626 (+ 5544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:18.693639 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:18.693659 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:18.693727 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:18.697701 (+ 3974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:18.698940 (+ 1239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:18.698951 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:18.698958 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:18.699195 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:18.699823 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:18.699829 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:18.699832 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:18.700530 (+ 698us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:18.700731 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:18.701822 (+ 1091us) server_negotiation.cc:299] Negotiation successful
0806 02:55:18.702315 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14771,"thread_start_us":15046,"threads_started":1}
I20260806 02:55:18.707914 13494 heartbeater.cc:347] Connected to a master server at 127.25.211.190:42641
I20260806 02:55:18.708307 13494 heartbeater.cc:464] Registering TS with master...
I20260806 02:55:18.709270 13494 heartbeater.cc:511] Master 127.25.211.190:42641 requested a full tablet report, sending...
W20260806 02:55:18.713646 13494 heartbeater.cc:644] Failed to heartbeat to 127.25.211.190:42641 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 02:55:18.713903 13494 heartbeater.cc:464] Registering TS with master...
I20260806 02:55:18.714234 13494 heartbeater.cc:511] Master 127.25.211.190:42641 requested a full tablet report, sending...
I20260806 02:55:18.715737 13494 heartbeater.cc:464] Registering TS with master...
I20260806 02:55:18.716041 13494 heartbeater.cc:511] Master 127.25.211.190:42641 requested a full tablet report, sending...
W20260806 02:55:18.717381 13494 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 02:55:19.057752 13495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:55:19.058059 13495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:55:19.058202 13495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:55:19.069643 13495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:55:19.069892 13495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:55:19.070218 13495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.130
I20260806 02:55:19.086992 13495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:37055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.25.211.148:60817/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab
--principal=kudu/127.25.211.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.130:35413
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.211.130
--webserver_port=37251
--tserver_master_addrs=127.25.211.190:42641
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.130
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:55:19.088985 13495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:55:19.090936 13495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:55:19.109256 13515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:19.109251 13518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:55:19.110729 13495 server_base.cc:1034] running on GCE node
W20260806 02:55:19.110860 13522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:55:19.111820 13495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:55:19.122866 13495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:55:19.130577 13495 hybrid_clock.cc:648] HybridClock initialized: now 1785984919130431 us; error 195 us; skew 500 ppm
I20260806 02:55:19.131199 13495 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:55:19 dist-test-slave-zj8q krb5kdc[11740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984919, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:55:19.135320 13495 init.cc:377] Logged in from keytab as kudu/127.25.211.130@KRBTEST.COM (short username kudu)
I20260806 02:55:19.137239 13495 webserver.cc:460] Webserver started at http://127.25.211.130:37251/ using document root <none> and password file <none>
I20260806 02:55:19.137992 13495 fs_manager.cc:359] Metadata directory not provided
I20260806 02:55:19.138142 13495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 02:55:19 dist-test-slave-zj8q krb5kdc[11740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984919, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.130@KRBTEST.COM for HTTP/127.25.211.148@KRBTEST.COM
I20260806 02:55:19.161813 13495 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 02:55:19.164909 13531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:19.166565 13495 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260806 02:55:19.166759 13495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-1/wal
uuid: "c1bd0f8a5f954652b72b6b38401d795b"
format_stamp: "Formatted at 2026-08-06 02:55:15 on dist-test-slave-zj8q"
server_key: "0ba2e77ffcb5708a0052edee23b41e03"
server_key_iv: "0c184d987321dd13670969f74e98c79e"
server_key_version: "kuduclusterkey@0"
I20260806 02:55:19.167821 13495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Aug 06 02:55:19 dist-test-slave-zj8q krb5kdc[11740](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1785984919, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:55:19.207796 13495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:55:19.215485 13495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:55:19.215885 13495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:55:19.219797 13495 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:55:19.227435 13495 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:55:19.227573 13495 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:19.227713 13495 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:55:19.227789 13495 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:19.278609 13495 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.130:35413
I20260806 02:55:19.278647 13687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.130:35413 every 8 connection(s)
I20260806 02:55:19.280303 13495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
I20260806 02:55:19.287571 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 13495
I20260806 02:55:19.288431 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.25.211.148:60817/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.131:32939
--local_ip_for_outbound_sockets=127.25.211.131
--tserver_master_addrs=127.25.211.190:42641
--webserver_port=45677
--webserver_interface=127.25.211.131
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab
--principal=kudu/127.25.211.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:37055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.25.211.148:60817/kms with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:55:19 dist-test-slave-zj8q krb5kdc[11740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984919, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.130@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:55:19.310125 13696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:19.283559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:40061 (local address 127.25.211.190:42641)
0806 02:55:19.284082 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:19.284089 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:19.285376 (+ 1287us) server_negotiation.cc:407] Connection header received
0806 02:55:19.288137 (+ 2761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:19.288145 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:19.288262 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:19.288348 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:19.290801 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:19.292051 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:19.294885 (+ 2834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:19.295547 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:19.295663 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:19.301598 (+ 5935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:19.301608 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:19.301613 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:19.301667 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:19.305513 (+ 3846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:19.306808 (+ 1295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:19.306816 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:19.306819 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:19.307002 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:19.307695 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:19.307702 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:19.307705 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:19.308132 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:19.308287 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:19.309458 (+ 1171us) server_negotiation.cc:299] Negotiation successful
0806 02:55:19.309740 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":248,"threads_started":1}
I20260806 02:55:19.311539 13697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:19.284174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:42641 (local address 127.25.211.130:40061)
0806 02:55:19.285112 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:19.285164 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 02:55:19.287739 (+ 2575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:55:19.288971 (+ 1232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:55:19.288983 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:55:19.289618 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:55:19.290543 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:19.290576 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:19.292318 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:19.292324 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:19.294619 (+ 2295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:19.294674 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:19.296074 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:19.296083 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:19.296451 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:19.297338 (+ 887us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:55:19.297373 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:55:19.301221 (+ 3848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:55:19.305757 (+ 4536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:19.305770 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:19.305793 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:19.306563 (+ 770us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:19.307180 (+ 617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:19.307185 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:19.307188 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:19.307517 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:19.308399 (+ 882us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:55:19.308409 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:55:19.308948 (+ 539us) client_negotiation.cc:769] Sending connection context
0806 02:55:19.309274 (+ 326us) client_negotiation.cc:240] Negotiation successful
0806 02:55:19.309980 (+ 706us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":307,"threads_started":1}
I20260806 02:55:19.313297 13690 heartbeater.cc:347] Connected to a master server at 127.25.211.190:42641
I20260806 02:55:19.313649 13690 heartbeater.cc:464] Registering TS with master...
I20260806 02:55:19.314530 13690 heartbeater.cc:511] Master 127.25.211.190:42641 requested a full tablet report, sending...
I20260806 02:55:19.316483 13213 ts_manager.cc:194] Registered new tserver with Master: c1bd0f8a5f954652b72b6b38401d795b (127.25.211.130:35413)
I20260806 02:55:19.318325 13213 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.211.130@KRBTEST.COM'} at 127.25.211.130:40061
Aug 06 02:55:19 dist-test-slave-zj8q krb5kdc[11740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984919, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.190@KRBTEST.COM for HTTP/127.25.211.148@KRBTEST.COM
I20260806 02:55:19.717792 13494 heartbeater.cc:464] Registering TS with master...
I20260806 02:55:19.718286 13494 heartbeater.cc:511] Master 127.25.211.190:42641 requested a full tablet report, sending...
I20260806 02:55:19.719676 13213 ts_manager.cc:194] Registered new tserver with Master: 372649a493804ce487faf5268d7fb947 (127.25.211.129:43113)
I20260806 02:55:19.721131 13213 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.211.129@KRBTEST.COM'} at 127.25.211.129:48767
W20260806 02:55:19.787974 13699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:55:19.788404 13699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:55:19.788630 13699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:55:19.798470 13699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:55:19.798615 13699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:55:19.798794 13699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.131
I20260806 02:55:19.809331 13699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:37055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.25.211.148:60817/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab
--principal=kudu/127.25.211.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.131:32939
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.211.131
--webserver_port=45677
--tserver_master_addrs=127.25.211.190:42641
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.131
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:55:19.810952 13699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:55:19.812363 13699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:55:19.828733 13708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:19.828697 13710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:19.834281 13712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:55:19.835481 13699 server_base.cc:1034] running on GCE node
I20260806 02:55:19.836373 13699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:55:19.837719 13699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:55:19.839010 13699 hybrid_clock.cc:648] HybridClock initialized: now 1785984919838933 us; error 57 us; skew 500 ppm
I20260806 02:55:19.839425 13699 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:55:19 dist-test-slave-zj8q krb5kdc[11740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984919, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:55:19.844312 13699 init.cc:377] Logged in from keytab as kudu/127.25.211.131@KRBTEST.COM (short username kudu)
I20260806 02:55:19.865198 13699 webserver.cc:460] Webserver started at http://127.25.211.131:45677/ using document root <none> and password file <none>
I20260806 02:55:19.866451 13699 fs_manager.cc:359] Metadata directory not provided
I20260806 02:55:19.866660 13699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 02:55:19 dist-test-slave-zj8q krb5kdc[11740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984919, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.131@KRBTEST.COM for HTTP/127.25.211.148@KRBTEST.COM
I20260806 02:55:19.895960 13699 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260806 02:55:19.899926 13726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:19.901841 13699 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:55:19.902056 13699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-2/wal
uuid: "f09305aa77aa4bec9462ba1ea4230291"
format_stamp: "Formatted at 2026-08-06 02:55:16 on dist-test-slave-zj8q"
server_key: "30e3e506f6c07fcdffb04cf6a1693a4a"
server_key_iv: "4df08539b89294be0ebb03ab9b0b47d1"
server_key_version: "kuduclusterkey@0"
I20260806 02:55:19.903462 13699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:55:19.925629 13699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:55:19.931161 13699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:55:19.931480 13699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:55:19.932765 13699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:55:19.934690 13699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:55:19.934813 13699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:19.934930 13699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:55:19.935011 13699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 02:55:19 dist-test-slave-zj8q krb5kdc[11740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984919, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.190@KRBTEST.COM for HTTP/127.25.211.148@KRBTEST.COM
I20260806 02:55:20.206442 13699 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.131:32939
I20260806 02:55:20.206508 13895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.131:32939 every 8 connection(s)
I20260806 02:55:20.208588 13699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
I20260806 02:55:20.213964 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 13699
Aug 06 02:55:20 dist-test-slave-zj8q krb5kdc[11740](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984919, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.131@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:55:20.251818 13899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:20.215197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53078 (local address 127.25.211.190:42641)
0806 02:55:20.227047 (+ 11850us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:20.227055 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:20.227084 (+ 29us) server_negotiation.cc:407] Connection header received
0806 02:55:20.227164 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:20.227170 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:20.227375 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:20.227532 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:20.231203 (+ 3671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:20.232398 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:20.235234 (+ 2836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:20.235924 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:20.236056 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:20.242446 (+ 6390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:20.242455 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:20.242460 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:20.242508 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:20.246821 (+ 4313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:20.248139 (+ 1318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:20.248146 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:20.248150 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:20.248380 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:20.248902 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:20.248907 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:20.248910 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:20.249370 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:20.251009 (+ 1639us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:20.251156 (+ 147us) server_negotiation.cc:299] Negotiation successful
0806 02:55:20.251412 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11674,"thread_start_us":906,"threads_started":1}
I20260806 02:55:20.255335 13898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:20.212260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:42641 (local address 127.25.211.131:38883)
0806 02:55:20.228337 (+ 16077us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:20.228410 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0806 02:55:20.230835 (+ 2425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:55:20.233570 (+ 2735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:55:20.233585 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:55:20.234357 (+ 772us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:55:20.236419 (+ 2062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:20.236452 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:20.238102 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:20.238107 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:20.240432 (+ 2325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:20.240463 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:20.241902 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:20.241908 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:20.242137 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:20.242984 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:55:20.243021 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:55:20.245419 (+ 2398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:55:20.249942 (+ 4523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:20.249956 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:20.250014 (+ 58us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:20.250789 (+ 775us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:20.251272 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:20.251277 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:20.251280 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:20.251557 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:20.252281 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:55:20.252290 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:55:20.252793 (+ 503us) client_negotiation.cc:769] Sending connection context
0806 02:55:20.253111 (+ 318us) client_negotiation.cc:240] Negotiation successful
0806 02:55:20.253743 (+ 632us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15702,"thread_start_us":15475,"threads_started":1}
I20260806 02:55:20.256531 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:20.212920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:38883 (local address 127.25.211.190:42641)
0806 02:55:20.216373 (+ 3453us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:20.216380 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:20.228544 (+ 12164us) server_negotiation.cc:407] Connection header received
0806 02:55:20.232624 (+ 4080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:20.232631 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:20.232746 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:20.232830 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:20.236668 (+ 3838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:20.237868 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:20.240637 (+ 2769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:20.241402 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:20.241552 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:20.245754 (+ 4202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:20.245763 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:20.245768 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:20.245829 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:20.249707 (+ 3878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:20.250970 (+ 1263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:20.250976 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:20.250979 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:20.251120 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:20.251707 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:20.251712 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:20.251715 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:20.252102 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:20.255784 (+ 3682us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:20.255976 (+ 192us) server_negotiation.cc:299] Negotiation successful
0806 02:55:20.256234 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3324,"thread_start_us":14047,"threads_started":1}
I20260806 02:55:20.257716 13896 heartbeater.cc:347] Connected to a master server at 127.25.211.190:42641
I20260806 02:55:20.258112 13896 heartbeater.cc:464] Registering TS with master...
I20260806 02:55:20.259162 13896 heartbeater.cc:511] Master 127.25.211.190:42641 requested a full tablet report, sending...
I20260806 02:55:20.260972 13213 ts_manager.cc:194] Registered new tserver with Master: f09305aa77aa4bec9462ba1ea4230291 (127.25.211.131:32939)
I20260806 02:55:20.261831 13213 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.211.131@KRBTEST.COM'} at 127.25.211.131:38883
I20260806 02:55:20.267668 26446 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:55:20.284391 13902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:20.270643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53080 (local address 127.25.211.190:42641)
0806 02:55:20.270808 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:20.270813 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:20.271275 (+ 462us) server_negotiation.cc:407] Connection header received
0806 02:55:20.271434 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:20.271438 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:20.271545 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:20.271674 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:20.273362 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:20.274135 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:20.275561 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:20.275971 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:20.276082 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:20.279014 (+ 2932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:20.279020 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:20.279023 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:20.279070 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:20.281680 (+ 2610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:20.282543 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:20.282547 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:20.282550 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:20.282676 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:20.283127 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:20.283132 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:20.283134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:20.283440 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:20.283641 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:20.283958 (+ 317us) server_negotiation.cc:299] Negotiation successful
0806 02:55:20.284154 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 02:55:20.289263 13213 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53080:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:55:20.296747 13213 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:55:20.323032 13690 heartbeater.cc:503] Master 127.25.211.190:42641 was elected leader, sending a full tablet report...
I20260806 02:55:20.328066 13929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:20.311800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38222 (local address 127.25.211.130:35413)
0806 02:55:20.319174 (+ 7374us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:20.319183 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:20.319212 (+ 29us) server_negotiation.cc:407] Connection header received
0806 02:55:20.319321 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:20.319328 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:20.319557 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:20.320216 (+ 659us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:55:20.320523 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:20.322129 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:20.325076 (+ 2947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:20.326664 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:20.326881 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:20.327077 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:20.327498 (+ 421us) server_negotiation.cc:299] Negotiation successful
0806 02:55:20.327683 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7214,"thread_start_us":7086,"threads_started":1}
I20260806 02:55:20.328328 13930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:20.312122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47510 (local address 127.25.211.129:43113)
0806 02:55:20.318885 (+ 6763us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:20.318893 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:20.318919 (+ 26us) server_negotiation.cc:407] Connection header received
0806 02:55:20.319003 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:20.319010 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:20.319232 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:20.320374 (+ 1142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:55:20.320632 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:20.321827 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:20.325431 (+ 3604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:20.326699 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:20.326904 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:20.327083 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:20.327863 (+ 780us) server_negotiation.cc:299] Negotiation successful
0806 02:55:20.328007 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6577,"thread_start_us":6394,"threads_started":1}
I20260806 02:55:20.328498 13936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:20.314261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43982 (local address 127.25.211.131:32939)
0806 02:55:20.317955 (+ 3694us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:20.317965 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:20.318576 (+ 611us) server_negotiation.cc:407] Connection header received
0806 02:55:20.318939 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:20.318948 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:20.319253 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:20.319424 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:55:20.320872 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:20.322450 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:20.325294 (+ 2844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:20.327009 (+ 1715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:20.327200 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:20.327365 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:20.327966 (+ 601us) server_negotiation.cc:299] Negotiation successful
0806 02:55:20.328137 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3517,"thread_start_us":3366,"threads_started":1}
I20260806 02:55:20.328816 13926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:20.311546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.130:35413 (local address 127.0.0.1:38222)
0806 02:55:20.314370 (+ 2824us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:20.314426 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 02:55:20.314615 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:55:20.319778 (+ 5163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:55:20.319782 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:55:20.319812 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:55:20.320345 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:20.320358 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:20.322278 (+ 1920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:20.322283 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:20.324678 (+ 2395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:20.324695 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:20.326920 (+ 2225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:20.326927 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:20.327144 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:20.327210 (+ 66us) client_negotiation.cc:769] Sending connection context
0806 02:55:20.327384 (+ 174us) client_negotiation.cc:240] Negotiation successful
0806 02:55:20.327627 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2449,"mutex_wait_us":55,"thread_start_us":3270,"threads_started":1}
I20260806 02:55:20.328779 13931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:20.311606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:43113 (local address 127.0.0.1:47510)
0806 02:55:20.317182 (+ 5576us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:20.317203 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 02:55:20.317334 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:55:20.319556 (+ 2222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:55:20.319563 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:55:20.319604 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:55:20.320153 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:20.320169 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:20.321967 (+ 1798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:20.321972 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:20.325242 (+ 3270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:20.325269 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:20.327279 (+ 2010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:20.327286 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:20.327479 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:20.327511 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 02:55:20.327588 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 02:55:20.327700 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5432,"mutex_wait_us":8,"thread_start_us":6578,"threads_started":1}
I20260806 02:55:20.330410 13928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:20.314119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.131:32939 (local address 127.0.0.1:43982)
0806 02:55:20.318451 (+ 4332us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:20.318471 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 02:55:20.318732 (+ 261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:55:20.319790 (+ 1058us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:55:20.319795 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:55:20.319840 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:55:20.320685 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:20.320697 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:20.322569 (+ 1872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:20.322575 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:20.324903 (+ 2328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:20.324919 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:20.327338 (+ 2419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:20.327346 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:20.327543 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:20.327577 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 02:55:20.327666 (+ 89us) client_negotiation.cc:240] Negotiation successful
0806 02:55:20.327806 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4220,"mutex_wait_us":239,"spinlock_wait_cycles":1408,"thread_start_us":2732,"threads_started":1}
I20260806 02:55:20.333034 13406 tablet_service.cc:1467] Processing CreateTablet for tablet 85407e2780944fd7bf87ecb8049f879e (DEFAULT_TABLE table=test-table [id=e575a90c88984fafb99e5c841b5d8d59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:55:20.333158 13600 tablet_service.cc:1467] Processing CreateTablet for tablet 85407e2780944fd7bf87ecb8049f879e (DEFAULT_TABLE table=test-table [id=e575a90c88984fafb99e5c841b5d8d59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:55:20.334543 13406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85407e2780944fd7bf87ecb8049f879e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:55:20.334618 13600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85407e2780944fd7bf87ecb8049f879e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:55:20.334491 13791 tablet_service.cc:1467] Processing CreateTablet for tablet 85407e2780944fd7bf87ecb8049f879e (DEFAULT_TABLE table=test-table [id=e575a90c88984fafb99e5c841b5d8d59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:55:20.335968 13791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85407e2780944fd7bf87ecb8049f879e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:55:20.349996 13938 tablet_bootstrap.cc:492] T 85407e2780944fd7bf87ecb8049f879e P 372649a493804ce487faf5268d7fb947: Bootstrap starting.
I20260806 02:55:20.352451 13939 tablet_bootstrap.cc:492] T 85407e2780944fd7bf87ecb8049f879e P c1bd0f8a5f954652b72b6b38401d795b: Bootstrap starting.
I20260806 02:55:20.355990 13938 tablet_bootstrap.cc:654] T 85407e2780944fd7bf87ecb8049f879e P 372649a493804ce487faf5268d7fb947: Neither blocks nor log segments found. Creating new log.
I20260806 02:55:20.356276 13939 tablet_bootstrap.cc:654] T 85407e2780944fd7bf87ecb8049f879e P c1bd0f8a5f954652b72b6b38401d795b: Neither blocks nor log segments found. Creating new log.
I20260806 02:55:20.357327 13940 tablet_bootstrap.cc:492] T 85407e2780944fd7bf87ecb8049f879e P f09305aa77aa4bec9462ba1ea4230291: Bootstrap starting.
I20260806 02:55:20.357488 13939 log.cc:824] T 85407e2780944fd7bf87ecb8049f879e P c1bd0f8a5f954652b72b6b38401d795b: Log is configured to *not* fsync() on all Append() calls
I20260806 02:55:20.357532 13938 log.cc:824] T 85407e2780944fd7bf87ecb8049f879e P 372649a493804ce487faf5268d7fb947: Log is configured to *not* fsync() on all Append() calls
I20260806 02:55:20.361081 13939 tablet_bootstrap.cc:492] T 85407e2780944fd7bf87ecb8049f879e P c1bd0f8a5f954652b72b6b38401d795b: No bootstrap required, opened a new log
I20260806 02:55:20.361428 13939 ts_tablet_manager.cc:1397] T 85407e2780944fd7bf87ecb8049f879e P c1bd0f8a5f954652b72b6b38401d795b: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260806 02:55:20.361900 13938 tablet_bootstrap.cc:492] T 85407e2780944fd7bf87ecb8049f879e P 372649a493804ce487faf5268d7fb947: No bootstrap required, opened a new log
I20260806 02:55:20.362095 13940 tablet_bootstrap.cc:654] T 85407e2780944fd7bf87ecb8049f879e P f09305aa77aa4bec9462ba1ea4230291: Neither blocks nor log segments found. Creating new log.
I20260806 02:55:20.362227 13938 ts_tablet_manager.cc:1397] T 85407e2780944fd7bf87ecb8049f879e P 372649a493804ce487faf5268d7fb947: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20260806 02:55:20.363204 13940 log.cc:824] T 85407e2780944fd7bf87ecb8049f879e P f09305aa77aa4bec9462ba1ea4230291: Log is configured to *not* fsync() on all Append() calls
I20260806 02:55:20.368896 13940 tablet_bootstrap.cc:492] T 85407e2780944fd7bf87ecb8049f879e P f09305aa77aa4bec9462ba1ea4230291: No bootstrap required, opened a new log
I20260806 02:55:20.369333 13940 ts_tablet_manager.cc:1397] T 85407e2780944fd7bf87ecb8049f879e P f09305aa77aa4bec9462ba1ea4230291: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.003s
I20260806 02:55:20.372257 13939 raft_consensus.cc:348] T 85407e2780944fd7bf87ecb8049f879e P c1bd0f8a5f954652b72b6b38401d795b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1bd0f8a5f954652b72b6b38401d795b" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 35413 } } peers { permanent_uuid: "372649a493804ce487faf5268d7fb947" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 43113 } } peers { permanent_uuid: "f09305aa77aa4bec9462ba1ea4230291" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 32939 } }
I20260806 02:55:20.372795 13939 raft_consensus.cc:374] T 85407e2780944fd7bf87ecb8049f879e P c1bd0f8a5f954652b72b6b38401d795b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:55:20.372965 13939 raft_consensus.cc:729] T 85407e2780944fd7bf87ecb8049f879e P c1bd0f8a5f954652b72b6b38401d795b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1bd0f8a5f954652b72b6b38401d795b, State: Initialized, Role: FOLLOWER
I20260806 02:55:20.373690 13939 consensus_queue.cc:260] T 85407e2780944fd7bf87ecb8049f879e P c1bd0f8a5f954652b72b6b38401d795b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1bd0f8a5f954652b72b6b38401d795b" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 35413 } } peers { permanent_uuid: "372649a493804ce487faf5268d7fb947" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 43113 } } peers { permanent_uuid: "f09305aa77aa4bec9462ba1ea4230291" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 32939 } }
I20260806 02:55:20.373591 13938 raft_consensus.cc:348] T 85407e2780944fd7bf87ecb8049f879e P 372649a493804ce487faf5268d7fb947 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1bd0f8a5f954652b72b6b38401d795b" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 35413 } } peers { permanent_uuid: "372649a493804ce487faf5268d7fb947" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 43113 } } peers { permanent_uuid: "f09305aa77aa4bec9462ba1ea4230291" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 32939 } }
I20260806 02:55:20.374059 13938 raft_consensus.cc:374] T 85407e2780944fd7bf87ecb8049f879e P 372649a493804ce487faf5268d7fb947 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:55:20.374202 13938 raft_consensus.cc:729] T 85407e2780944fd7bf87ecb8049f879e P 372649a493804ce487faf5268d7fb947 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 372649a493804ce487faf5268d7fb947, State: Initialized, Role: FOLLOWER
I20260806 02:55:20.375041 13938 consensus_queue.cc:260] T 85407e2780944fd7bf87ecb8049f879e P 372649a493804ce487faf5268d7fb947 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1bd0f8a5f954652b72b6b38401d795b" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 35413 } } peers { permanent_uuid: "372649a493804ce487faf5268d7fb947" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 43113 } } peers { permanent_uuid: "f09305aa77aa4bec9462ba1ea4230291" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 32939 } }
I20260806 02:55:20.375698 13939 ts_tablet_manager.cc:1428] T 85407e2780944fd7bf87ecb8049f879e P c1bd0f8a5f954652b72b6b38401d795b: Time spent starting tablet: real 0.014s user 0.013s sys 0.002s
I20260806 02:55:20.378255 13940 raft_consensus.cc:348] T 85407e2780944fd7bf87ecb8049f879e P f09305aa77aa4bec9462ba1ea4230291 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1bd0f8a5f954652b72b6b38401d795b" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 35413 } } peers { permanent_uuid: "372649a493804ce487faf5268d7fb947" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 43113 } } peers { permanent_uuid: "f09305aa77aa4bec9462ba1ea4230291" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 32939 } }
I20260806 02:55:20.378760 13940 raft_consensus.cc:374] T 85407e2780944fd7bf87ecb8049f879e P f09305aa77aa4bec9462ba1ea4230291 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:55:20.378861 13940 raft_consensus.cc:729] T 85407e2780944fd7bf87ecb8049f879e P f09305aa77aa4bec9462ba1ea4230291 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f09305aa77aa4bec9462ba1ea4230291, State: Initialized, Role: FOLLOWER
I20260806 02:55:20.379637 13940 consensus_queue.cc:260] T 85407e2780944fd7bf87ecb8049f879e P f09305aa77aa4bec9462ba1ea4230291 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1bd0f8a5f954652b72b6b38401d795b" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 35413 } } peers { permanent_uuid: "372649a493804ce487faf5268d7fb947" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 43113 } } peers { permanent_uuid: "f09305aa77aa4bec9462ba1ea4230291" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 32939 } }
I20260806 02:55:20.381191 13896 heartbeater.cc:503] Master 127.25.211.190:42641 was elected leader, sending a full tablet report...
I20260806 02:55:20.381614 13940 ts_tablet_manager.cc:1428] T 85407e2780944fd7bf87ecb8049f879e P f09305aa77aa4bec9462ba1ea4230291: Time spent starting tablet: real 0.012s user 0.008s sys 0.005s
I20260806 02:55:20.382898 13494 heartbeater.cc:503] Master 127.25.211.190:42641 was elected leader, sending a full tablet report...
I20260806 02:55:20.383358 13938 ts_tablet_manager.cc:1428] T 85407e2780944fd7bf87ecb8049f879e P 372649a493804ce487faf5268d7fb947: Time spent starting tablet: real 0.021s user 0.006s sys 0.012s
W20260806 02:55:20.426538 13496 tablet.cc:2369] T 85407e2780944fd7bf87ecb8049f879e P 372649a493804ce487faf5268d7fb947: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:55:20.429137 13946 raft_consensus.cc:482] T 85407e2780944fd7bf87ecb8049f879e P 372649a493804ce487faf5268d7fb947 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:55:20.429435 13946 raft_consensus.cc:504] T 85407e2780944fd7bf87ecb8049f879e P 372649a493804ce487faf5268d7fb947 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1bd0f8a5f954652b72b6b38401d795b" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 35413 } } peers { permanent_uuid: "372649a493804ce487faf5268d7fb947" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 43113 } } peers { permanent_uuid: "f09305aa77aa4bec9462ba1ea4230291" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 32939 } }
I20260806 02:55:20.431205 13946 leader_election.cc:288] T 85407e2780944fd7bf87ecb8049f879e P 372649a493804ce487faf5268d7fb947 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1bd0f8a5f954652b72b6b38401d795b (127.25.211.130:35413), f09305aa77aa4bec9462ba1ea4230291 (127.25.211.131:32939)
I20260806 02:55:20.440341 13929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:20.431651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:58119 (local address 127.25.211.130:35413)
0806 02:55:20.431873 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:20.431879 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:20.432600 (+ 721us) server_negotiation.cc:407] Connection header received
0806 02:55:20.432835 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:20.432840 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:20.432911 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:20.433003 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:55:20.433927 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:20.435062 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:20.437935 (+ 2873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:20.439244 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:20.439428 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:20.439510 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:20.439938 (+ 428us) server_negotiation.cc:299] Negotiation successful
0806 02:55:20.440080 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260806 02:55:20.440341 13936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:20.431724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:36449 (local address 127.25.211.131:32939)
0806 02:55:20.431888 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:20.431893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:20.432627 (+ 734us) server_negotiation.cc:407] Connection header received
0806 02:55:20.432815 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:20.432820 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:20.432917 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:20.433002 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:55:20.433926 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:20.435062 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:20.437935 (+ 2873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:20.439225 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:20.439428 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:20.439510 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:20.439938 (+ 428us) server_negotiation.cc:299] Negotiation successful
0806 02:55:20.440080 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 02:55:20.440538 13950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:20.431526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.130:35413 (local address 127.25.211.129:58119)
0806 02:55:20.432472 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:20.432492 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 02:55:20.432705 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:55:20.433036 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:55:20.433040 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:55:20.433095 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:55:20.433759 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:20.433771 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:20.435205 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:20.435208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:20.437754 (+ 2546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:20.437774 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:20.439473 (+ 1699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:20.439484 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:20.439744 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:20.439800 (+ 56us) client_negotiation.cc:769] Sending connection context
0806 02:55:20.439902 (+ 102us) client_negotiation.cc:240] Negotiation successful
0806 02:55:20.440098 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":807,"mutex_wait_us":164,"spinlock_wait_cycles":72320,"thread_start_us":571,"threads_started":1}
I20260806 02:55:20.440578 13949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:20.431618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.131:32939 (local address 127.25.211.129:36449)
0806 02:55:20.432528 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:20.432541 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:55:20.432663 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:55:20.433036 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:55:20.433040 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:55:20.433091 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:55:20.433698 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:20.433732 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:20.435196 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:20.435202 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:20.437754 (+ 2552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:20.437774 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:20.439501 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:20.439506 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:20.439752 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:20.439803 (+ 51us) client_negotiation.cc:769] Sending connection context
0806 02:55:20.439902 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 02:55:20.440048 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":813,"mutex_wait_us":211,"spinlock_wait_cycles":18176,"thread_start_us":469,"threads_started":1}
I20260806 02:55:20.441699 13823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85407e2780944fd7bf87ecb8049f879e" candidate_uuid: "372649a493804ce487faf5268d7fb947" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f09305aa77aa4bec9462ba1ea4230291" is_pre_election: true
I20260806 02:55:20.441699 13628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85407e2780944fd7bf87ecb8049f879e" candidate_uuid: "372649a493804ce487faf5268d7fb947" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1bd0f8a5f954652b72b6b38401d795b" is_pre_election: true
I20260806 02:55:20.442263 13823 raft_consensus.cc:2454] T 85407e2780944fd7bf87ecb8049f879e P f09305aa77aa4bec9462ba1ea4230291 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 372649a493804ce487faf5268d7fb947 in term 0.
I20260806 02:55:20.442263 13628 raft_consensus.cc:2454] T 85407e2780944fd7bf87ecb8049f879e P c1bd0f8a5f954652b72b6b38401d795b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 372649a493804ce487faf5268d7fb947 in term 0.
I20260806 02:55:20.442946 13338 leader_election.cc:302] T 85407e2780944fd7bf87ecb8049f879e P 372649a493804ce487faf5268d7fb947 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 372649a493804ce487faf5268d7fb947, c1bd0f8a5f954652b72b6b38401d795b; no voters:
I20260806 02:55:20.443421 13946 raft_consensus.cc:2781] T 85407e2780944fd7bf87ecb8049f879e P 372649a493804ce487faf5268d7fb947 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:55:20.443563 13946 raft_consensus.cc:482] T 85407e2780944fd7bf87ecb8049f879e P 372649a493804ce487faf5268d7fb947 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:55:20.443660 13946 raft_consensus.cc:3037] T 85407e2780944fd7bf87ecb8049f879e P 372649a493804ce487faf5268d7fb947 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:55:20.445888 13946 raft_consensus.cc:504] T 85407e2780944fd7bf87ecb8049f879e P 372649a493804ce487faf5268d7fb947 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1bd0f8a5f954652b72b6b38401d795b" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 35413 } } peers { permanent_uuid: "372649a493804ce487faf5268d7fb947" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 43113 } } peers { permanent_uuid: "f09305aa77aa4bec9462ba1ea4230291" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 32939 } }
I20260806 02:55:20.446563 13946 leader_election.cc:288] T 85407e2780944fd7bf87ecb8049f879e P 372649a493804ce487faf5268d7fb947 [CANDIDATE]: Term 1 election: Requested vote from peers c1bd0f8a5f954652b72b6b38401d795b (127.25.211.130:35413), f09305aa77aa4bec9462ba1ea4230291 (127.25.211.131:32939)
I20260806 02:55:20.447054 13628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85407e2780944fd7bf87ecb8049f879e" candidate_uuid: "372649a493804ce487faf5268d7fb947" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1bd0f8a5f954652b72b6b38401d795b"
I20260806 02:55:20.447060 13823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85407e2780944fd7bf87ecb8049f879e" candidate_uuid: "372649a493804ce487faf5268d7fb947" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f09305aa77aa4bec9462ba1ea4230291"
I20260806 02:55:20.447237 13628 raft_consensus.cc:3037] T 85407e2780944fd7bf87ecb8049f879e P c1bd0f8a5f954652b72b6b38401d795b [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:55:20.447244 13823 raft_consensus.cc:3037] T 85407e2780944fd7bf87ecb8049f879e P f09305aa77aa4bec9462ba1ea4230291 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:55:20.449431 13823 raft_consensus.cc:2454] T 85407e2780944fd7bf87ecb8049f879e P f09305aa77aa4bec9462ba1ea4230291 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 372649a493804ce487faf5268d7fb947 in term 1.
I20260806 02:55:20.449440 13628 raft_consensus.cc:2454] T 85407e2780944fd7bf87ecb8049f879e P c1bd0f8a5f954652b72b6b38401d795b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 372649a493804ce487faf5268d7fb947 in term 1.
I20260806 02:55:20.449852 13338 leader_election.cc:302] T 85407e2780944fd7bf87ecb8049f879e P 372649a493804ce487faf5268d7fb947 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 372649a493804ce487faf5268d7fb947, c1bd0f8a5f954652b72b6b38401d795b; no voters:
I20260806 02:55:20.450120 13946 raft_consensus.cc:2781] T 85407e2780944fd7bf87ecb8049f879e P 372649a493804ce487faf5268d7fb947 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:55:20.450609 13946 raft_consensus.cc:686] T 85407e2780944fd7bf87ecb8049f879e P 372649a493804ce487faf5268d7fb947 [term 1 LEADER]: Becoming Leader. State: Replica: 372649a493804ce487faf5268d7fb947, State: Running, Role: LEADER
I20260806 02:55:20.451099 13946 consensus_queue.cc:237] T 85407e2780944fd7bf87ecb8049f879e P 372649a493804ce487faf5268d7fb947 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1bd0f8a5f954652b72b6b38401d795b" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 35413 } } peers { permanent_uuid: "372649a493804ce487faf5268d7fb947" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 43113 } } peers { permanent_uuid: "f09305aa77aa4bec9462ba1ea4230291" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 32939 } }
I20260806 02:55:20.456101 13213 catalog_manager.cc:5495] T 85407e2780944fd7bf87ecb8049f879e P 372649a493804ce487faf5268d7fb947 reported cstate change: term changed from 0 to 1, leader changed from <none> to 372649a493804ce487faf5268d7fb947 (127.25.211.129). New cstate: current_term: 1 leader_uuid: "372649a493804ce487faf5268d7fb947" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1bd0f8a5f954652b72b6b38401d795b" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 35413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "372649a493804ce487faf5268d7fb947" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 43113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f09305aa77aa4bec9462ba1ea4230291" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 32939 } health_report { overall_health: UNKNOWN } } }
W20260806 02:55:20.477276 13897 tablet.cc:2369] T 85407e2780944fd7bf87ecb8049f879e P f09305aa77aa4bec9462ba1ea4230291: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:55:20.479838 13930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:20.474494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47526 (local address 127.25.211.129:43113)
0806 02:55:20.474671 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:20.474676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:20.474693 (+ 17us) server_negotiation.cc:407] Connection header received
0806 02:55:20.474877 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:20.474882 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:20.474960 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:20.475090 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:55:20.475416 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:20.476149 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:20.477558 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:20.478204 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:20.478314 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:20.478816 (+ 502us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:55:20.479343 (+ 527us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:55:20.479449 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:20.479543 (+ 94us) server_negotiation.cc:299] Negotiation successful
0806 02:55:20.479620 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 02:55:20.488175 13823 raft_consensus.cc:1261] T 85407e2780944fd7bf87ecb8049f879e P f09305aa77aa4bec9462ba1ea4230291 [term 1 FOLLOWER]: Refusing update from remote peer 372649a493804ce487faf5268d7fb947: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:55:20.488171 13628 raft_consensus.cc:1261] T 85407e2780944fd7bf87ecb8049f879e P c1bd0f8a5f954652b72b6b38401d795b [term 1 FOLLOWER]: Refusing update from remote peer 372649a493804ce487faf5268d7fb947: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:55:20.489154 13951 consensus_queue.cc:1035] T 85407e2780944fd7bf87ecb8049f879e P 372649a493804ce487faf5268d7fb947 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f09305aa77aa4bec9462ba1ea4230291" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 32939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:55:20.489507 13946 consensus_queue.cc:1035] T 85407e2780944fd7bf87ecb8049f879e P 372649a493804ce487faf5268d7fb947 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1bd0f8a5f954652b72b6b38401d795b" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 35413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:55:20.515071 13213 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:55:20.515339 13213 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:55:20.519120 13213 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 107613bd63d14cd1a485cbab9a3cb60a: Sending DeleteTablet for 3 replicas of tablet 85407e2780944fd7bf87ecb8049f879e
I20260806 02:55:20.520318 13600 tablet_service.cc:1514] Processing DeleteTablet for tablet 85407e2780944fd7bf87ecb8049f879e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:55:20 UTC) from {username='kudu', principal='kudu/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:38222
I20260806 02:55:20.520354 13791 tablet_service.cc:1514] Processing DeleteTablet for tablet 85407e2780944fd7bf87ecb8049f879e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:55:20 UTC) from {username='kudu', principal='kudu/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:43982
I20260806 02:55:20.521128 13406 tablet_service.cc:1514] Processing DeleteTablet for tablet 85407e2780944fd7bf87ecb8049f879e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:55:20 UTC) from {username='kudu', principal='kudu/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:47510
I20260806 02:55:20.527685 13971 tablet_replica.cc:330] T 85407e2780944fd7bf87ecb8049f879e P c1bd0f8a5f954652b72b6b38401d795b: stopping tablet replica
I20260806 02:55:20.528227 13971 raft_consensus.cc:2229] T 85407e2780944fd7bf87ecb8049f879e P c1bd0f8a5f954652b72b6b38401d795b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:55:20.528789 13971 raft_consensus.cc:2258] T 85407e2780944fd7bf87ecb8049f879e P c1bd0f8a5f954652b72b6b38401d795b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:55:20.529534 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 13268
I20260806 02:55:20.529626 13973 tablet_replica.cc:330] T 85407e2780944fd7bf87ecb8049f879e P f09305aa77aa4bec9462ba1ea4230291: stopping tablet replica
I20260806 02:55:20.530997 13973 raft_consensus.cc:2229] T 85407e2780944fd7bf87ecb8049f879e P f09305aa77aa4bec9462ba1ea4230291 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:55:20.531512 13973 raft_consensus.cc:2258] T 85407e2780944fd7bf87ecb8049f879e P f09305aa77aa4bec9462ba1ea4230291 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:55:20.532961 13971 ts_tablet_manager.cc:1905] T 85407e2780944fd7bf87ecb8049f879e P c1bd0f8a5f954652b72b6b38401d795b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:55:20.533708 13973 ts_tablet_manager.cc:1905] T 85407e2780944fd7bf87ecb8049f879e P f09305aa77aa4bec9462ba1ea4230291: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:55:20.540735 13973 ts_tablet_manager.cc:1918] T 85407e2780944fd7bf87ecb8049f879e P f09305aa77aa4bec9462ba1ea4230291: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:55:20.540977 13973 log.cc:1196] T 85407e2780944fd7bf87ecb8049f879e P f09305aa77aa4bec9462ba1ea4230291: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-2/wal/wals/85407e2780944fd7bf87ecb8049f879e
I20260806 02:55:20.541555 13973 ts_tablet_manager.cc:1939] T 85407e2780944fd7bf87ecb8049f879e P f09305aa77aa4bec9462ba1ea4230291: Deleting consensus metadata
I20260806 02:55:20.543699 13193 catalog_manager.cc:4841] TS f09305aa77aa4bec9462ba1ea4230291 (127.25.211.131:32939): tablet 85407e2780944fd7bf87ecb8049f879e (table test-table [id=e575a90c88984fafb99e5c841b5d8d59]) successfully deleted
I20260806 02:55:20.544965 13971 ts_tablet_manager.cc:1918] T 85407e2780944fd7bf87ecb8049f879e P c1bd0f8a5f954652b72b6b38401d795b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:55:20.545259 13971 log.cc:1196] T 85407e2780944fd7bf87ecb8049f879e P c1bd0f8a5f954652b72b6b38401d795b: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1785984707115702-26446-0/minicluster-data/ts-1/wal/wals/85407e2780944fd7bf87ecb8049f879e
I20260806 02:55:20.547051 13971 ts_tablet_manager.cc:1939] T 85407e2780944fd7bf87ecb8049f879e P c1bd0f8a5f954652b72b6b38401d795b: Deleting consensus metadata
I20260806 02:55:20.548553 13930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:20.531445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47538 (local address 127.25.211.129:43113)
0806 02:55:20.531627 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:20.531632 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:20.531655 (+ 23us) server_negotiation.cc:407] Connection header received
0806 02:55:20.531722 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:20.531727 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:20.531829 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:20.531946 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:20.533628 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:20.534897 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:20.536689 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:20.537346 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:20.537490 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:20.541080 (+ 3590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:20.541096 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:20.541117 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:20.541173 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:20.545201 (+ 4028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:20.546102 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:20.546113 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:20.546117 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:20.546352 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:20.546732 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:20.546737 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:20.546740 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:20.547252 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:20.547583 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:20.547905 (+ 322us) server_negotiation.cc:299] Negotiation successful
0806 02:55:20.548150 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 02:55:20.549542 13487 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:55:20.550405 13191 catalog_manager.cc:4841] TS c1bd0f8a5f954652b72b6b38401d795b (127.25.211.130:35413): tablet 85407e2780944fd7bf87ecb8049f879e (table test-table [id=e575a90c88984fafb99e5c841b5d8d59]) successfully deleted
I20260806 02:55:20.650094 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 13268
I20260806 02:55:20.650336 13974 tablet_replica.cc:330] T 85407e2780944fd7bf87ecb8049f879e P 372649a493804ce487faf5268d7fb947: stopping tablet replica
W20260806 02:55:20.672693 13192 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.211.129:43113 (error 108)
I20260806 02:55:20.673394 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 13495
I20260806 02:55:20.674620 13928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:20.674124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:43113 (local address 127.0.0.1:47544)
0806 02:55:20.674323 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:20.674447 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.129:43113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260806 02:55:20.674861 13192 catalog_manager.cc:4584] TS 372649a493804ce487faf5268d7fb947 (127.25.211.129:43113): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 85407e2780944fd7bf87ecb8049f879e: Network error: Client connection negotiation failed: client connection to 127.25.211.129:43113: connect: Connection refused (error 111)
I20260806 02:55:20.687804 13929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:20.674266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38230 (local address 127.25.211.130:35413)
0806 02:55:20.674449 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:20.674455 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:20.674476 (+ 21us) server_negotiation.cc:407] Connection header received
0806 02:55:20.674623 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:20.674629 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:20.674712 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:20.674827 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:20.676510 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:20.677287 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:20.678741 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:20.679174 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:20.679290 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:20.681942 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:20.681958 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:20.681976 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:20.682014 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:20.684874 (+ 2860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:20.685677 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:20.685686 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:20.685688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:20.685834 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:20.686233 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:20.686237 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:20.686239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:20.686608 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:20.686906 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:20.687210 (+ 304us) server_negotiation.cc:299] Negotiation successful
0806 02:55:20.687432 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260806 02:55:20.688498 13682 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:55:20.728652 13928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:20.728158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:43113 (local address 127.0.0.1:47548)
0806 02:55:20.728365 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:20.728479 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.129:43113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260806 02:55:20.729710 13928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:20.729374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:43113 (local address 127.0.0.1:47556)
0806 02:55:20.729509 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:20.729600 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.129:43113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260806 02:55:20.779074 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 13495
I20260806 02:55:20.790211 13928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:20.789625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:43113 (local address 127.0.0.1:47560)
0806 02:55:20.789862 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:20.789973 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.129:43113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260806 02:55:20.791327 13928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:20.790928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:43113 (local address 127.0.0.1:47570)
0806 02:55:20.791090 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:20.791182 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.129:43113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260806 02:55:20.804445 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 13699
I20260806 02:55:20.819033 13936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:20.805398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43994 (local address 127.25.211.131:32939)
0806 02:55:20.805626 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:20.805633 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:20.805657 (+ 24us) server_negotiation.cc:407] Connection header received
0806 02:55:20.805728 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:20.805734 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:20.805812 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:20.805899 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:20.807289 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:20.808066 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:20.809607 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:20.810085 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:20.810273 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:20.812964 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:20.812980 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:20.812998 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:20.813038 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:20.815916 (+ 2878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:20.816711 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:20.816718 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:20.816720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:20.816870 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:20.817261 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:20.817265 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:20.817268 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:20.817751 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:20.818071 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:20.818479 (+ 408us) server_negotiation.cc:299] Negotiation successful
0806 02:55:20.818705 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 02:55:20.819725 13888 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:55:20.871796 13928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:20.871345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:43113 (local address 127.0.0.1:47578)
0806 02:55:20.871551 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:20.871640 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.129:43113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260806 02:55:20.872813 13928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:20.872410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:43113 (local address 127.0.0.1:47588)
0806 02:55:20.872602 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:20.872695 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.129:43113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260806 02:55:20.915220 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 13699
I20260806 02:55:20.938671 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 13169
I20260806 02:55:20.939872 13259 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:55:21.054394 13928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:21.053691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:43113 (local address 127.0.0.1:47598)
0806 02:55:21.053934 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:21.054087 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.129:43113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260806 02:55:21.054729 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 13169
I20260806 02:55:21.084419 26446 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260806 02:55:21.430387 26446 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260806 02:55:21.430624 26446 mini_ranger.cc:66] Stopping Ranger...
I20260806 02:55:21.776422 26446 mini_ranger.cc:68] Stopped Ranger
2026-08-06 02:55:21.776 UTC [11782] LOG: received smart shutdown request
2026-08-06 02:55:21.779 UTC [11782] LOG: background worker "logical replication launcher" (PID 11797) exited with exit code 1
2026-08-06 02:55:21.779 UTC [11790] LOG: shutting down
2026-08-06 02:55:21.941 UTC [11782] LOG: database system is shut down
2026-08-06T02:55:21Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (56732 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:55:22 dist-test-slave-zj8q krb5kdc[13997](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:55:22 dist-test-slave-zj8q krb5kdc[13997](info): set up 2 sockets
Aug 06 02:55:22 dist-test-slave-zj8q krb5kdc[13997](info): commencing operation
krb5kdc: starting...
W20260806 02:55:23.276422 26446 mini_kdc.cc:121] Time spent starting KDC: real 1.173s user 0.001s sys 0.039s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Aug 06 02:55:23 dist-test-slave-zj8q krb5kdc[13997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:55:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:55:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.211.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab.
Entry for principal kudu/127.25.211.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab.
I20260806 02:55:23.570132 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:36995
--webserver_interface=127.25.211.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:38637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.211.190:36995,127.25.211.189:33187,127.25.211.188:43461
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:55:23.918944 14017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:55:23.919261 14017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:55:23.919394 14017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:55:23.926833 14017 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:55:23.926960 14017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:55:23.927030 14017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:55:23.927109 14017 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:55:23.927168 14017 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:55:23.937199 14017 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:38637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.211.190:36995,127.25.211.189:33187,127.25.211.188:43461
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:36995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.211.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:55:23.938706 14017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:55:23.940316 14017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:55:23.952570 14024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:23.952596 14027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:23.952801 14032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:55:23.953385 14017 server_base.cc:1034] running on GCE node
I20260806 02:55:23.954246 14017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:55:23.955806 14017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:55:23.957057 14017 hybrid_clock.cc:648] HybridClock initialized: now 1785984923957002 us; error 44 us; skew 500 ppm
I20260806 02:55:23.957437 14017 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:55:23 dist-test-slave-zj8q krb5kdc[13997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984923, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:55:23.964876 14017 init.cc:377] Logged in from keytab as kudu/127.25.211.190@KRBTEST.COM (short username kudu)
I20260806 02:55:23.966650 14017 webserver.cc:460] Webserver started at http://127.25.211.190:42023/ using document root <none> and password file <none>
I20260806 02:55:23.967402 14017 fs_manager.cc:359] Metadata directory not provided
I20260806 02:55:23.967532 14017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:55:23.967855 14017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:55:23.971014 14017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/minicluster-data/master-0/data/instance:
uuid: "62db4ddd40b64bb29919b6bb37253582"
format_stamp: "Formatted at 2026-08-06 02:55:23 on dist-test-slave-zj8q"
server_key: "f0ab40d57bf39e8476d2617882e13345"
server_key_iv: "4817589cbcf7b8ba365836bbabdd26c5"
server_key_version: "encryptionkey@0"
I20260806 02:55:23.971851 14017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/minicluster-data/master-0/wal/instance:
uuid: "62db4ddd40b64bb29919b6bb37253582"
format_stamp: "Formatted at 2026-08-06 02:55:23 on dist-test-slave-zj8q"
server_key: "f0ab40d57bf39e8476d2617882e13345"
server_key_iv: "4817589cbcf7b8ba365836bbabdd26c5"
server_key_version: "encryptionkey@0"
I20260806 02:55:23.977276 14017 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260806 02:55:23.980867 14043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:23.982244 14017 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260806 02:55:23.982426 14017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/minicluster-data/master-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/minicluster-data/master-0/wal
uuid: "62db4ddd40b64bb29919b6bb37253582"
format_stamp: "Formatted at 2026-08-06 02:55:23 on dist-test-slave-zj8q"
server_key: "f0ab40d57bf39e8476d2617882e13345"
server_key_iv: "4817589cbcf7b8ba365836bbabdd26c5"
server_key_version: "encryptionkey@0"
I20260806 02:55:23.982649 14017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:55:23.997393 14017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:55:24.001281 14017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:55:24.001536 14017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:55:24.020675 14017 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.190:36995
I20260806 02:55:24.020710 14122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.190:36995 every 8 connection(s)
I20260806 02:55:24.022107 14017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
I20260806 02:55:24.025859 14123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:55:24.030462 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 14017
I20260806 02:55:24.030678 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/minicluster-data/master-0/wal/instance
I20260806 02:55:24.031329 26446 external_mini_cluster.cc:1428] Setting key da816aff51d9b4ae5cf84b52a8cb196f
I20260806 02:55:24.032518 14123 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.211.190" port: 36995 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:55:24 dist-test-slave-zj8q krb5kdc[13997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984923, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.190@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:55:24.054957 14126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:24.035682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33812 (local address 127.25.211.190:36995)
0806 02:55:24.037408 (+ 1726us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:24.037416 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:24.037473 (+ 57us) server_negotiation.cc:407] Connection header received
0806 02:55:24.039086 (+ 1613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:24.039092 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:24.039325 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:24.039502 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:24.040816 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:24.042028 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:24.043431 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:24.043990 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:24.044156 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:24.047514 (+ 3358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:24.047524 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:24.047538 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:24.047569 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:24.050503 (+ 2934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:24.051227 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:24.051235 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:24.051237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:24.051406 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:24.051796 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:24.051799 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:24.051802 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:24.052338 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:24.052543 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:24.053070 (+ 527us) server_negotiation.cc:299] Negotiation successful
0806 02:55:24.053411 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1514,"thread_start_us":1231,"threads_started":1}
I20260806 02:55:24.054991 14125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:24.035682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:36995 (local address 127.0.0.1:33812)
0806 02:55:24.036559 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:24.036616 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 02:55:24.038712 (+ 2096us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:55:24.039546 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:55:24.039553 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:55:24.040283 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:55:24.040626 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:24.040643 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:24.042207 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:24.042212 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:24.043299 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:24.043312 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:24.044240 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:24.044247 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:24.044404 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:24.044982 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:55:24.045005 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:55:24.047285 (+ 2280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:55:24.050645 (+ 3360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:24.050658 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:24.050672 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:24.051083 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:24.051521 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:24.051526 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:24.051530 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:24.051687 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:24.052476 (+ 789us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:55:24.052484 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:55:24.052703 (+ 219us) client_negotiation.cc:769] Sending connection context
0806 02:55:24.052922 (+ 219us) client_negotiation.cc:240] Negotiation successful
0806 02:55:24.053550 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"mutex_wait_us":130,"thread_start_us":368,"threads_started":1}
WARNING: no policy specified for kudu/127.25.211.189@KRBTEST.COM; defaulting to no policy
I20260806 02:55:24.058145 14123 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.211.189" port: 33187 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.189@KRBTEST.COM" created.
I20260806 02:55:24.059275 14125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:24.058847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.189:33187 (local address 127.0.0.1:34050)
0806 02:55:24.059021 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:24.059146 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.189:33187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260806 02:55:24.059504 14048 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.211.189:33187: connect: Connection refused (error 111)
I20260806 02:55:24.061129 14125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:24.060767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.189:33187 (local address 127.0.0.1:34062)
0806 02:55:24.060911 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:24.060982 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.189:33187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260806 02:55:24.061313 14123 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.211.189:33187: Network error: Client connection negotiation failed: client connection to 127.25.211.189:33187: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.189.keytab.
Entry for principal kudu/127.25.211.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.189.keytab.
I20260806 02:55:24.091125 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.189:33187
--webserver_interface=127.25.211.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.189.keytab
--principal=kudu/127.25.211.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:38637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.211.190:36995,127.25.211.189:33187,127.25.211.188:43461
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:55:24.110711 14123 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.211.189" port: 33187 } attempt: 1
I20260806 02:55:24.112088 14125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:24.111526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.189:33187 (local address 127.0.0.1:34078)
0806 02:55:24.111764 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:24.111903 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.189:33187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260806 02:55:24.113157 14125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:24.112750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.189:33187 (local address 127.0.0.1:34092)
0806 02:55:24.112895 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:24.112984 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.189:33187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260806 02:55:24.113355 14123 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.211.189:33187: Network error: Client connection negotiation failed: client connection to 127.25.211.189:33187: connect: Connection refused (error 111)
I20260806 02:55:24.181670 14123 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.211.189" port: 33187 } attempt: 2
I20260806 02:55:24.183125 14125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:24.182491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.189:33187 (local address 127.0.0.1:34102)
0806 02:55:24.182804 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:24.182931 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.189:33187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260806 02:55:24.184306 14125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:24.183862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.189:33187 (local address 127.0.0.1:34110)
0806 02:55:24.184045 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:24.184138 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.189:33187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260806 02:55:24.184547 14123 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.211.189:33187: Network error: Client connection negotiation failed: client connection to 127.25.211.189:33187: connect: Connection refused (error 111)
I20260806 02:55:24.275847 14123 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.211.189" port: 33187 } attempt: 3
I20260806 02:55:24.277423 14125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:24.276779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.189:33187 (local address 127.0.0.1:34126)
0806 02:55:24.277072 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:24.277223 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.189:33187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260806 02:55:24.278733 14125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:24.278221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.189:33187 (local address 127.0.0.1:34130)
0806 02:55:24.278398 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:24.278499 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.189:33187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260806 02:55:24.278959 14123 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.211.189:33187: Network error: Client connection negotiation failed: client connection to 127.25.211.189:33187: connect: Connection refused (error 111)
I20260806 02:55:24.422286 14123 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.211.189" port: 33187 } attempt: 4
I20260806 02:55:24.423650 14125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:24.423026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.189:33187 (local address 127.0.0.1:34136)
0806 02:55:24.423362 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:24.423472 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.189:33187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171}
I20260806 02:55:24.424695 14125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:24.424279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.189:33187 (local address 127.0.0.1:34140)
0806 02:55:24.424434 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:24.424536 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.189:33187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260806 02:55:24.424873 14123 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.211.189:33187: Network error: Client connection negotiation failed: client connection to 127.25.211.189:33187: connect: Connection refused (error 111)
W20260806 02:55:24.439837 14132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:55:24.440132 14132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:55:24.440222 14132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:55:24.447496 14132 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:55:24.447619 14132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:55:24.447681 14132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:55:24.447742 14132 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:55:24.447785 14132 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:55:24.457849 14132 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:38637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.25.211.190:36995,127.25.211.189:33187,127.25.211.188:43461
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.189.keytab
--principal=kudu/127.25.211.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.189:33187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.25.211.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:55:24.459319 14132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:55:24.460635 14132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:55:24.472651 14145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:24.472723 14149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:24.472672 14146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:55:24.473557 14132 server_base.cc:1034] running on GCE node
I20260806 02:55:24.474496 14132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:55:24.475925 14132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:55:24.477128 14132 hybrid_clock.cc:648] HybridClock initialized: now 1785984924477078 us; error 36 us; skew 500 ppm
I20260806 02:55:24.477465 14132 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:55:24 dist-test-slave-zj8q krb5kdc[13997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984924, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:55:24.480634 14132 init.cc:377] Logged in from keytab as kudu/127.25.211.189@KRBTEST.COM (short username kudu)
I20260806 02:55:24.482144 14132 webserver.cc:460] Webserver started at http://127.25.211.189:40069/ using document root <none> and password file <none>
I20260806 02:55:24.482966 14132 fs_manager.cc:359] Metadata directory not provided
I20260806 02:55:24.483073 14132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:55:24.483336 14132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:55:24.486506 14132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/minicluster-data/master-1/data/instance:
uuid: "95d94d857c6948dfa46874b44cc0bd75"
format_stamp: "Formatted at 2026-08-06 02:55:24 on dist-test-slave-zj8q"
server_key: "51be493c732fbbb39b865c1ab2c58f81"
server_key_iv: "72ad33d2dcc693e1ad7235bcc5304a55"
server_key_version: "encryptionkey@0"
I20260806 02:55:24.487383 14132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/minicluster-data/master-1/wal/instance:
uuid: "95d94d857c6948dfa46874b44cc0bd75"
format_stamp: "Formatted at 2026-08-06 02:55:24 on dist-test-slave-zj8q"
server_key: "51be493c732fbbb39b865c1ab2c58f81"
server_key_iv: "72ad33d2dcc693e1ad7235bcc5304a55"
server_key_version: "encryptionkey@0"
I20260806 02:55:24.493239 14132 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 02:55:24.496991 14160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:24.498476 14132 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 02:55:24.498654 14132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/minicluster-data/master-1/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/minicluster-data/master-1/wal
uuid: "95d94d857c6948dfa46874b44cc0bd75"
format_stamp: "Formatted at 2026-08-06 02:55:24 on dist-test-slave-zj8q"
server_key: "51be493c732fbbb39b865c1ab2c58f81"
server_key_iv: "72ad33d2dcc693e1ad7235bcc5304a55"
server_key_version: "encryptionkey@0"
I20260806 02:55:24.498867 14132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:55:24.521490 14132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:55:24.525751 14132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:55:24.526038 14132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:55:24.544866 14132 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.189:33187
I20260806 02:55:24.544888 14237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.189:33187 every 8 connection(s)
I20260806 02:55:24.546442 14132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/minicluster-data/master-1/data/info.pb
I20260806 02:55:24.550635 14238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:55:24.551088 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 14132
I20260806 02:55:24.551265 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/minicluster-data/master-1/wal/instance
I20260806 02:55:24.551765 26446 external_mini_cluster.cc:1428] Setting key 7b94631659059199b1ac763098efa5ab
I20260806 02:55:24.558018 14238 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.211.190" port: 36995 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:55:24 dist-test-slave-zj8q krb5kdc[13997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984924, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.189@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.211.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.188@KRBTEST.COM" created.
I20260806 02:55:24.582906 14240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:24.560773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33814 (local address 127.25.211.190:36995)
0806 02:55:24.563135 (+ 2362us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:24.563142 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:24.563931 (+ 789us) server_negotiation.cc:407] Connection header received
0806 02:55:24.566967 (+ 3036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:24.566974 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:24.567074 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:24.567178 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:24.569060 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:24.569872 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:24.571648 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:24.572090 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:24.572211 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:24.576278 (+ 4067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:24.576284 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:24.576288 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:24.576327 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:24.579367 (+ 3040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:24.580189 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:24.580194 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:24.580197 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:24.580335 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:24.580810 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:24.580814 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:24.580816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:24.581256 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:24.581377 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:24.582251 (+ 874us) server_negotiation.cc:299] Negotiation successful
0806 02:55:24.582526 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2208,"thread_start_us":2063,"threads_started":1}
I20260806 02:55:24.583762 14241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:24.561481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:36995 (local address 127.0.0.1:33814)
0806 02:55:24.563735 (+ 2254us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:24.563802 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0806 02:55:24.566546 (+ 2744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:55:24.567507 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:55:24.567527 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:55:24.568169 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:55:24.568833 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:24.568853 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:24.570069 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:24.570074 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:24.571464 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:24.571481 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:24.572344 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:24.572352 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:24.572595 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:24.573310 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:55:24.573335 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:55:24.575954 (+ 2619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:55:24.579530 (+ 3576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:24.579539 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:24.579554 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:24.580028 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:24.580477 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:24.580480 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:24.580483 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:24.580673 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:24.581402 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:55:24.581411 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:55:24.581825 (+ 414us) client_negotiation.cc:769] Sending connection context
0806 02:55:24.582083 (+ 258us) client_negotiation.cc:240] Negotiation successful
0806 02:55:24.582566 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1930,"thread_start_us":1665,"threads_started":1}
I20260806 02:55:24.585696 14238 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.211.189" port: 33187 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:55:24 dist-test-slave-zj8q krb5kdc[13997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984924, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.189@KRBTEST.COM for kudu/127.25.211.189@KRBTEST.COM
I20260806 02:55:24.602322 14241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:24.586421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.189:33187 (local address 127.0.0.1:34146)
0806 02:55:24.586670 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:24.586686 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:55:24.586812 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:55:24.588172 (+ 1360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:55:24.588178 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:55:24.588669 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:55:24.588865 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:24.588877 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:24.590681 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:24.590687 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:24.591827 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:24.591844 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:24.593057 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:24.593064 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:24.593270 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:24.593950 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:55:24.593967 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:55:24.596398 (+ 2431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:55:24.599698 (+ 3300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:24.599702 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:24.599706 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:24.600199 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:24.600576 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:24.600580 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:24.600583 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:24.600765 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:24.601366 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:55:24.601371 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:55:24.601513 (+ 142us) client_negotiation.cc:769] Sending connection context
0806 02:55:24.601647 (+ 134us) client_negotiation.cc:240] Negotiation successful
0806 02:55:24.601944 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137}
I20260806 02:55:24.602438 14247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:24.586806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34146 (local address 127.25.211.189:33187)
0806 02:55:24.587616 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:24.587624 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:24.587658 (+ 34us) server_negotiation.cc:407] Connection header received
0806 02:55:24.587720 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:24.587728 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:24.587972 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:24.588124 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:24.589040 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:24.590506 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:24.591980 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:24.592881 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:24.593608 (+ 727us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:24.596655 (+ 3047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:24.596669 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:24.596686 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:24.596723 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:24.599537 (+ 2814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:24.600331 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:24.600338 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:24.600340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:24.600472 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:24.600860 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:24.600863 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:24.600865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:24.601231 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:24.601399 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:24.601778 (+ 379us) server_negotiation.cc:299] Negotiation successful
0806 02:55:24.602079 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":82,"server-negotiator.queue_time_us":615,"thread_start_us":454,"threads_started":1}
I20260806 02:55:24.604925 14238 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.211.188" port: 43461 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:55:24.606202 14241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:24.605585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:43461 (local address 127.0.0.1:42990)
0806 02:55:24.605793 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:24.605951 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.188:43461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260806 02:55:24.606508 14161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.211.188:43461: connect: Connection refused (error 111)
I20260806 02:55:24.608201 14241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:24.607768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:43461 (local address 127.0.0.1:42998)
0806 02:55:24.607961 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:24.608061 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.188:43461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260806 02:55:24.608438 14238 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.211.188:43461: Network error: Client connection negotiation failed: client connection to 127.25.211.188:43461: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.188.keytab.
Entry for principal kudu/127.25.211.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.188.keytab.
I20260806 02:55:24.616999 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.188:43461
--webserver_interface=127.25.211.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.188.keytab
--principal=kudu/127.25.211.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:38637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.211.190:36995,127.25.211.189:33187,127.25.211.188:43461
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:55:24.657892 14238 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.211.188" port: 43461 } attempt: 1
I20260806 02:55:24.659229 14241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:24.658648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:43461 (local address 127.0.0.1:43008)
0806 02:55:24.658897 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:24.659033 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.188:43461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260806 02:55:24.660349 14241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:24.659949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:43461 (local address 127.0.0.1:43016)
0806 02:55:24.660122 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:24.660211 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.188:43461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260806 02:55:24.660606 14238 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.211.188:43461: Network error: Client connection negotiation failed: client connection to 127.25.211.188:43461: connect: Connection refused (error 111)
I20260806 02:55:24.724189 14123 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.211.189" port: 33187 } attempt: 5
I20260806 02:55:24.728860 14238 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.211.188" port: 43461 } attempt: 2
I20260806 02:55:24.730001 14241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:24.729467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:43461 (local address 127.0.0.1:43028)
0806 02:55:24.729679 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:24.729799 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.188:43461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260806 02:55:24.731570 14241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:24.731140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:43461 (local address 127.0.0.1:43030)
0806 02:55:24.731320 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:24.731414 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.188:43461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260806 02:55:24.731885 14238 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.211.188:43461: Network error: Client connection negotiation failed: client connection to 127.25.211.188:43461: connect: Connection refused (error 111)
Aug 06 02:55:24 dist-test-slave-zj8q krb5kdc[13997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984923, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.190@KRBTEST.COM for kudu/127.25.211.189@KRBTEST.COM
I20260806 02:55:24.738780 14125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:24.724993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.189:33187 (local address 127.0.0.1:34160)
0806 02:55:24.725296 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:24.725320 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 02:55:24.725461 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:55:24.725962 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:55:24.725966 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:55:24.726430 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:55:24.726618 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:24.726629 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:24.728085 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:24.728089 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:24.729153 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:24.729171 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:24.730073 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:24.730079 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:24.730321 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:24.730994 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:55:24.731013 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:55:24.733303 (+ 2290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:55:24.736335 (+ 3032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:24.736340 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:24.736344 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:24.736728 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:24.737099 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:24.737104 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:24.737108 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:24.737254 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:24.737782 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:55:24.737787 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:55:24.737935 (+ 148us) client_negotiation.cc:769] Sending connection context
0806 02:55:24.738058 (+ 123us) client_negotiation.cc:240] Negotiation successful
0806 02:55:24.738420 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139}
I20260806 02:55:24.738780 14247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:24.725343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34160 (local address 127.25.211.189:33187)
0806 02:55:24.725563 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:24.725569 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:24.725593 (+ 24us) server_negotiation.cc:407] Connection header received
0806 02:55:24.725678 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:24.725684 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:24.725775 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:24.725883 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:24.726808 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:24.727903 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:24.729343 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:24.729909 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:24.730027 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:24.733498 (+ 3471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:24.733503 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:24.733507 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:24.733541 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:24.736155 (+ 2614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:24.736850 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:24.736855 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:24.736857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:24.736984 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:24.737362 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:24.737365 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:24.737368 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:24.737664 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:24.737799 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:24.738250 (+ 451us) server_negotiation.cc:299] Negotiation successful
0806 02:55:24.738465 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 02:55:24.740388 14123 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.211.188" port: 43461 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:55:24.741575 14125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:24.741166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:43461 (local address 127.0.0.1:43032)
0806 02:55:24.741346 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:24.741427 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.188:43461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260806 02:55:24.742612 14125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:24.742283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:43461 (local address 127.0.0.1:43048)
0806 02:55:24.742409 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:24.742499 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.188:43461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260806 02:55:24.742852 14123 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.211.188:43461: Network error: Client connection negotiation failed: client connection to 127.25.211.188:43461: connect: Connection refused (error 111)
I20260806 02:55:24.794080 14123 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.211.188" port: 43461 } attempt: 1
I20260806 02:55:24.795388 14125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:24.794856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:43461 (local address 127.0.0.1:43060)
0806 02:55:24.795107 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:24.795221 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.188:43461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260806 02:55:24.796447 14125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:24.796072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:43461 (local address 127.0.0.1:43062)
0806 02:55:24.796249 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:24.796326 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.188:43461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260806 02:55:24.796700 14123 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.211.188:43461: Network error: Client connection negotiation failed: client connection to 127.25.211.188:43461: connect: Connection refused (error 111)
I20260806 02:55:24.823132 14238 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.211.188" port: 43461 } attempt: 3
I20260806 02:55:24.824438 14241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:24.823930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:43461 (local address 127.0.0.1:43064)
0806 02:55:24.824160 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:24.824260 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.188:43461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260806 02:55:24.825613 14241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:24.825144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:43461 (local address 127.0.0.1:43074)
0806 02:55:24.825383 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:24.825469 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.188:43461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
W20260806 02:55:24.825908 14238 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.211.188:43461: Network error: Client connection negotiation failed: client connection to 127.25.211.188:43461: connect: Connection refused (error 111)
I20260806 02:55:24.864940 14123 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.211.188" port: 43461 } attempt: 2
I20260806 02:55:24.866354 14125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:24.865689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:43461 (local address 127.0.0.1:43086)
0806 02:55:24.865977 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:24.866094 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.188:43461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260806 02:55:24.867487 14125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:24.867071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:43461 (local address 127.0.0.1:43092)
0806 02:55:24.867247 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:24.867340 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.188:43461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260806 02:55:24.867753 14123 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.211.188:43461: Network error: Client connection negotiation failed: client connection to 127.25.211.188:43461: connect: Connection refused (error 111)
W20260806 02:55:24.964377 14250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:55:24.964685 14250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:55:24.964828 14250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260806 02:55:24.969201 14238 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.211.188" port: 43461 } attempt: 4
I20260806 02:55:24.970503 14241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:24.969956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:43461 (local address 127.0.0.1:43096)
0806 02:55:24.970240 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:24.970338 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.188:43461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260806 02:55:24.971575 14241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:24.971219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:43461 (local address 127.0.0.1:43102)
0806 02:55:24.971360 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:24.971434 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.188:43461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260806 02:55:24.971801 14238 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.211.188:43461: Network error: Client connection negotiation failed: client connection to 127.25.211.188:43461: connect: Connection refused (error 111)
W20260806 02:55:24.972775 14250 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:55:24.972884 14250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:55:24.972955 14250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:55:24.973033 14250 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:55:24.973097 14250 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:55:24.973980 14123 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.211.188" port: 43461 } attempt: 3
I20260806 02:55:24.975129 14125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:24.974631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:43461 (local address 127.0.0.1:43104)
0806 02:55:24.974838 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:24.974945 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.188:43461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260806 02:55:24.976272 14125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:24.975839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:43461 (local address 127.0.0.1:43112)
0806 02:55:24.976051 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:24.976138 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.188:43461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20260806 02:55:24.976511 14123 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.211.188:43461: Network error: Client connection negotiation failed: client connection to 127.25.211.188:43461: connect: Connection refused (error 111)
I20260806 02:55:24.983978 14250 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:38637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.25.211.190:36995,127.25.211.189:33187,127.25.211.188:43461
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.188.keytab
--principal=kudu/127.25.211.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.188:43461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.25.211.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:55:24.985443 14250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:55:24.986933 14250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:55:24.999042 14257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:24.999025 14260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:24.999094 14258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:55:25.000259 14250 server_base.cc:1034] running on GCE node
I20260806 02:55:25.001055 14250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:55:25.002600 14250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:55:25.003820 14250 hybrid_clock.cc:648] HybridClock initialized: now 1785984925003766 us; error 44 us; skew 500 ppm
I20260806 02:55:25.004190 14250 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:55:25 dist-test-slave-zj8q krb5kdc[13997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984925, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:55:25.007494 14250 init.cc:377] Logged in from keytab as kudu/127.25.211.188@KRBTEST.COM (short username kudu)
I20260806 02:55:25.009007 14250 webserver.cc:460] Webserver started at http://127.25.211.188:45557/ using document root <none> and password file <none>
I20260806 02:55:25.009737 14250 fs_manager.cc:359] Metadata directory not provided
I20260806 02:55:25.009859 14250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:55:25.010191 14250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:55:25.013449 14250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/minicluster-data/master-2/data/instance:
uuid: "d05d486fd0b84750b254692b1c6c362a"
format_stamp: "Formatted at 2026-08-06 02:55:25 on dist-test-slave-zj8q"
server_key: "fb4f683cb515316e0da6e19bce084b93"
server_key_iv: "60b9c68c449944d06aac75226fa19e61"
server_key_version: "encryptionkey@0"
I20260806 02:55:25.014361 14250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/minicluster-data/master-2/wal/instance:
uuid: "d05d486fd0b84750b254692b1c6c362a"
format_stamp: "Formatted at 2026-08-06 02:55:25 on dist-test-slave-zj8q"
server_key: "fb4f683cb515316e0da6e19bce084b93"
server_key_iv: "60b9c68c449944d06aac75226fa19e61"
server_key_version: "encryptionkey@0"
I20260806 02:55:25.019667 14250 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260806 02:55:25.023270 14270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:25.024655 14250 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260806 02:55:25.024838 14250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/minicluster-data/master-2/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/minicluster-data/master-2/wal
uuid: "d05d486fd0b84750b254692b1c6c362a"
format_stamp: "Formatted at 2026-08-06 02:55:25 on dist-test-slave-zj8q"
server_key: "fb4f683cb515316e0da6e19bce084b93"
server_key_iv: "60b9c68c449944d06aac75226fa19e61"
server_key_version: "encryptionkey@0"
I20260806 02:55:25.025020 14250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:55:25.036564 14250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:55:25.041177 14250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:55:25.041501 14250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:55:25.060421 14250 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.188:43461
I20260806 02:55:25.060456 14343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.188:43461 every 8 connection(s)
I20260806 02:55:25.061858 14250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/minicluster-data/master-2/data/info.pb
I20260806 02:55:25.065606 14344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:55:25.065954 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 14250
I20260806 02:55:25.066134 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACert.1785984707115702-26446-0/minicluster-data/master-2/wal/instance
I20260806 02:55:25.066640 26446 external_mini_cluster.cc:1428] Setting key d16542169f3f1b44278ccbb1e42261b9
I20260806 02:55:25.073913 14344 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.211.190" port: 36995 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:55:25 dist-test-slave-zj8q krb5kdc[13997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.188@KRBTEST.COM
Aug 06 02:55:25 dist-test-slave-zj8q krb5kdc[13997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984925, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.188@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:55:25.089963 14351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:25.068406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43124 (local address 127.25.211.188:43461)
0806 02:55:25.069181 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:25.069193 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:25.069232 (+ 39us) server_negotiation.cc:407] Connection header received
0806 02:55:25.071568 (+ 2336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:25.071590 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:25.071964 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:25.072332 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:25.073839 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:25.075220 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:25.076856 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:25.077721 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:25.077864 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:25.082481 (+ 4617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:25.082490 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:25.082504 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:25.082551 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:25.085749 (+ 3198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:25.086658 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:25.086665 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:25.086673 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:25.086847 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:25.087266 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:25.087272 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:25.087276 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:25.087803 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:25.088068 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:25.088559 (+ 491us) server_negotiation.cc:299] Negotiation successful
0806 02:55:25.088989 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":326,"threads_started":1}
I20260806 02:55:25.092242 14240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:25.076367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33824 (local address 127.25.211.190:36995)
0806 02:55:25.076588 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:25.076596 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:25.077223 (+ 627us) server_negotiation.cc:407] Connection header received
0806 02:55:25.077394 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:25.077400 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:25.077476 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:25.077569 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:25.079034 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:25.079755 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:25.081436 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:25.081890 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:25.081994 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:25.085709 (+ 3715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:25.085714 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:25.085718 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:25.085750 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:25.088579 (+ 2829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:25.089364 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:25.089370 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:25.089373 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:25.089500 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:25.089840 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:25.089843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:25.089845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:25.091059 (+ 1214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:25.091202 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:25.091691 (+ 489us) server_negotiation.cc:299] Negotiation successful
0806 02:55:25.091910 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 02:55:25.092278 14352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:25.076331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:36995 (local address 127.0.0.1:33824)
0806 02:55:25.077085 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:25.077122 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 02:55:25.077279 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:55:25.077654 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:55:25.077662 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:55:25.078532 (+ 870us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:55:25.078847 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:25.078861 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:25.079922 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:25.079927 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:25.081275 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:25.081288 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:25.082140 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:25.082148 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:25.082352 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:25.082985 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:55:25.083006 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:55:25.085447 (+ 2441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:55:25.088715 (+ 3268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:25.088727 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:25.088734 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:25.089251 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:25.089607 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:25.089612 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:25.089615 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:25.089739 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:25.091197 (+ 1458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:55:25.091204 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:55:25.091374 (+ 170us) client_negotiation.cc:769] Sending connection context
0806 02:55:25.091547 (+ 173us) client_negotiation.cc:240] Negotiation successful
0806 02:55:25.091865 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":381,"threads_started":1}
I20260806 02:55:25.093709 14344 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.211.189" port: 33187 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:55:25 dist-test-slave-zj8q krb5kdc[13997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984925, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.188@KRBTEST.COM for kudu/127.25.211.189@KRBTEST.COM
Aug 06 02:55:25 dist-test-slave-zj8q krb5kdc[13997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:55:25.108296 14352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:25.094358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.189:33187 (local address 127.0.0.1:34172)
0806 02:55:25.094558 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:25.094575 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:55:25.094662 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:55:25.095036 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:55:25.095040 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:55:25.095479 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:55:25.095644 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:25.095656 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:25.096886 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:25.096892 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:25.098006 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:25.098022 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:25.099037 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:25.099043 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:25.099247 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:25.099741 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:55:25.099760 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:55:25.102090 (+ 2330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:55:25.105450 (+ 3360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:25.105457 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:25.105461 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:25.106018 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:25.106607 (+ 589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:25.106611 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:25.106613 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:25.106765 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:25.107335 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:55:25.107339 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:55:25.107500 (+ 161us) client_negotiation.cc:769] Sending connection context
0806 02:55:25.107666 (+ 166us) client_negotiation.cc:240] Negotiation successful
0806 02:55:25.107980 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112}
I20260806 02:55:25.108309 14247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:25.094528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34172 (local address 127.25.211.189:33187)
0806 02:55:25.094757 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:25.094762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:25.094781 (+ 19us) server_negotiation.cc:407] Connection header received
0806 02:55:25.094838 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:25.094843 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:25.094914 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:25.094990 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:25.095779 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:25.096726 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:25.098238 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:25.098890 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:25.099009 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:25.102354 (+ 3345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:25.102360 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:25.102363 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:25.102399 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:25.105291 (+ 2892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:25.106228 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:25.106237 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:25.106241 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:25.106484 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:25.106874 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:25.106878 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:25.106881 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:25.107213 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:25.107368 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:25.107752 (+ 384us) server_negotiation.cc:299] Negotiation successful
0806 02:55:25.107922 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260806 02:55:25.109431 14240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:25.093091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33832 (local address 127.25.211.190:36995)
0806 02:55:25.093245 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:25.093251 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:25.093266 (+ 15us) server_negotiation.cc:407] Connection header received
0806 02:55:25.093318 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:25.093323 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:25.093388 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:25.093488 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:25.094948 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:25.095944 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:25.097471 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:25.098186 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:25.098286 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:25.103069 (+ 4783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:25.103076 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:25.103080 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:25.103125 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:25.105988 (+ 2863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:25.106972 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:25.106976 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:25.106978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:25.107100 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:25.107515 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:25.107520 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:25.107523 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:25.107868 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:25.108138 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:25.108863 (+ 725us) server_negotiation.cc:299] Negotiation successful
0806 02:55:25.109084 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 02:55:25.109606 14344 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.211.188" port: 43461 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:55:25 dist-test-slave-zj8q krb5kdc[13997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984925, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.188@KRBTEST.COM for kudu/127.25.211.188@KRBTEST.COM
Aug 06 02:55:25 dist-test-slave-zj8q krb5kdc[13997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.189@KRBTEST.COM
I20260806 02:55:25.124631 14352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:25.110315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:43461 (local address 127.0.0.1:43126)
0806 02:55:25.110601 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:25.110638 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 02:55:25.110736 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:55:25.111398 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:55:25.111403 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:55:25.111873 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:55:25.112051 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:25.112063 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:25.113009 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:25.113014 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:25.114379 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:25.114391 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:25.115389 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:25.115393 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:25.115528 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:25.116082 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:55:25.116105 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:55:25.118462 (+ 2357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:55:25.121354 (+ 2892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:25.121359 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:25.121364 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:25.121852 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:25.122812 (+ 960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:25.122817 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:25.122821 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:25.123001 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:25.123606 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:55:25.123611 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:55:25.123777 (+ 166us) client_negotiation.cc:769] Sending connection context
0806 02:55:25.123946 (+ 169us) client_negotiation.cc:240] Negotiation successful
0806 02:55:25.124277 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170}
I20260806 02:55:25.125598 14351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:25.110484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43126 (local address 127.25.211.188:43461)
0806 02:55:25.110659 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:25.110664 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:25.110679 (+ 15us) server_negotiation.cc:407] Connection header received
0806 02:55:25.110897 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:25.110903 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:25.110979 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:25.111073 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:25.112190 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:25.112878 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:25.114523 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:25.115224 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:25.115350 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:25.118651 (+ 3301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:25.118656 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:25.118660 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:25.118690 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:25.121224 (+ 2534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:25.122537 (+ 1313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:25.122543 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:25.122547 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:25.122697 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:25.123117 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:25.123121 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:25.123124 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:25.123476 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:25.123628 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:25.124115 (+ 487us) server_negotiation.cc:299] Negotiation successful
0806 02:55:25.125256 (+ 1141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":846,"server-negotiator.queue_time_us":44}
I20260806 02:55:25.127786 14247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:25.111267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34182 (local address 127.25.211.189:33187)
0806 02:55:25.111466 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:25.111470 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:25.111487 (+ 17us) server_negotiation.cc:407] Connection header received
0806 02:55:25.111525 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:25.111529 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:25.111580 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:25.111658 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:25.113051 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:25.114129 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:25.115838 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:25.116441 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:25.116569 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:25.121002 (+ 4433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:25.121007 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:25.121011 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:25.121045 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:25.124184 (+ 3139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:25.125041 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:25.125044 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:25.125047 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:25.125185 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:25.125644 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:25.125649 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:25.125653 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:25.125992 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:25.126399 (+ 407us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:25.126592 (+ 193us) server_negotiation.cc:299] Negotiation successful
0806 02:55:25.126811 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260806 02:55:25.132748 14344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d05d486fd0b84750b254692b1c6c362a: Bootstrap starting.
I20260806 02:55:25.136978 14344 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d05d486fd0b84750b254692b1c6c362a: Neither blocks nor log segments found. Creating new log.
I20260806 02:55:25.138432 14344 log.cc:824] T 00000000000000000000000000000000 P d05d486fd0b84750b254692b1c6c362a: Log is configured to *not* fsync() on all Append() calls
I20260806 02:55:25.140084 26446 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:55:25.142004 14344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d05d486fd0b84750b254692b1c6c362a: No bootstrap required, opened a new log
I20260806 02:55:25.153712 14123 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.211.188" port: 43461 } attempt: 4
I20260806 02:55:25.154055 14344 raft_consensus.cc:348] T 00000000000000000000000000000000 P d05d486fd0b84750b254692b1c6c362a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62db4ddd40b64bb29919b6bb37253582" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 36995 } } peers { permanent_uuid: "95d94d857c6948dfa46874b44cc0bd75" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 33187 } } peers { permanent_uuid: "d05d486fd0b84750b254692b1c6c362a" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 43461 } }
I20260806 02:55:25.154750 14344 raft_consensus.cc:374] T 00000000000000000000000000000000 P d05d486fd0b84750b254692b1c6c362a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:55:25.154953 14344 raft_consensus.cc:729] T 00000000000000000000000000000000 P d05d486fd0b84750b254692b1c6c362a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d05d486fd0b84750b254692b1c6c362a, State: Initialized, Role: FOLLOWER
I20260806 02:55:25.155694 14344 consensus_queue.cc:260] T 00000000000000000000000000000000 P d05d486fd0b84750b254692b1c6c362a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62db4ddd40b64bb29919b6bb37253582" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 36995 } } peers { permanent_uuid: "95d94d857c6948dfa46874b44cc0bd75" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 33187 } } peers { permanent_uuid: "d05d486fd0b84750b254692b1c6c362a" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 43461 } }
I20260806 02:55:25.156731 14364 sys_catalog.cc:455] T 00000000000000000000000000000000 P d05d486fd0b84750b254692b1c6c362a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62db4ddd40b64bb29919b6bb37253582" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 36995 } } peers { permanent_uuid: "95d94d857c6948dfa46874b44cc0bd75" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 33187 } } peers { permanent_uuid: "d05d486fd0b84750b254692b1c6c362a" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 43461 } } }
I20260806 02:55:25.157236 14364 sys_catalog.cc:458] T 00000000000000000000000000000000 P d05d486fd0b84750b254692b1c6c362a [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:55:25.158373 14344 sys_catalog.cc:564] T 00000000000000000000000000000000 P d05d486fd0b84750b254692b1c6c362a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:55:25.172973 14247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:25.148667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34184 (local address 127.25.211.189:33187)
0806 02:55:25.148942 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:25.148948 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:25.148966 (+ 18us) server_negotiation.cc:407] Connection header received
0806 02:55:25.149051 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:25.149057 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:25.149130 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:25.149205 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:25.152257 (+ 3052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:25.153339 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:25.156030 (+ 2691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:25.156779 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:25.156892 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:25.161409 (+ 4517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:25.161419 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:25.161425 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:25.161482 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:25.169010 (+ 7528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:25.170261 (+ 1251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:25.170268 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:25.170271 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:25.170464 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:25.170978 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:25.170983 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:25.170986 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:25.171410 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:25.171663 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:25.172301 (+ 638us) server_negotiation.cc:299] Negotiation successful
0806 02:55:25.172585 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148}
Aug 06 02:55:25 dist-test-slave-zj8q krb5kdc[13997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984923, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.190@KRBTEST.COM for kudu/127.25.211.188@KRBTEST.COM
I20260806 02:55:25.183538 14351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:25.149374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43138 (local address 127.25.211.188:43461)
0806 02:55:25.149569 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:25.149575 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:25.154559 (+ 4984us) server_negotiation.cc:407] Connection header received
0806 02:55:25.160362 (+ 5803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:25.160370 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:25.160484 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:25.160616 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:25.162769 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:25.163892 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:25.166076 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:25.166805 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:25.166943 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:25.173274 (+ 6331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:25.173350 (+ 76us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:25.173358 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:25.173416 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:25.177792 (+ 4376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:25.179853 (+ 2061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:25.179860 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:25.179865 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:25.180089 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:25.180618 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:25.180623 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:25.180627 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:25.181139 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:25.181440 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:25.182771 (+ 1331us) server_negotiation.cc:299] Negotiation successful
0806 02:55:25.183063 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 02:55:25.187146 14240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:25.147780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33846 (local address 127.25.211.190:36995)
0806 02:55:25.147990 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:25.147997 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:25.148258 (+ 261us) server_negotiation.cc:407] Connection header received
0806 02:55:25.148463 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:25.148469 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:25.148569 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:25.148668 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:25.152251 (+ 3583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:25.153398 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:25.165272 (+ 11874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:25.166041 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:25.166190 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:25.170508 (+ 4318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:25.170516 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:25.170521 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:25.170568 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:25.177492 (+ 6924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:25.183275 (+ 5783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:25.183282 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:25.183285 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:25.183458 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:25.183966 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:25.183972 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:25.183976 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:25.184454 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:25.184768 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:25.186447 (+ 1679us) server_negotiation.cc:299] Negotiation successful
0806 02:55:25.186752 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260806 02:55:25.196354 14125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:25.154388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:43461 (local address 127.0.0.1:43142)
0806 02:55:25.154808 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:25.154827 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:55:25.154939 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:55:25.171327 (+ 16388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:55:25.171333 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:55:25.171810 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:55:25.172030 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:25.172045 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:25.174131 (+ 2086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:25.174138 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:25.175678 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:25.175699 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:25.176817 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:25.176824 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:25.177075 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:25.178690 (+ 1615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:55:25.178714 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:55:25.181670 (+ 2956us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:55:25.191632 (+ 9962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:25.191639 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:25.191643 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:25.192228 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:25.194255 (+ 2027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:25.194261 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:25.194265 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:25.194467 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:25.195313 (+ 846us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:55:25.195319 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:55:25.195489 (+ 170us) client_negotiation.cc:769] Sending connection context
0806 02:55:25.195633 (+ 144us) client_negotiation.cc:240] Negotiation successful
0806 02:55:25.195951 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296}
I20260806 02:55:25.197718 14362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:25.154805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43142 (local address 127.25.211.188:43461)
0806 02:55:25.169924 (+ 15119us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:25.169930 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:25.169960 (+ 30us) server_negotiation.cc:407] Connection header received
0806 02:55:25.170022 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:25.170027 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:25.170114 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:25.172205 (+ 2091us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:25.172536 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:25.173914 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:25.175898 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:25.176618 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:25.177388 (+ 770us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:25.181969 (+ 4581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:25.181978 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:25.181984 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:25.182027 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:25.191398 (+ 9371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:25.192392 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:25.192396 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:25.192398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:25.192526 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:25.194634 (+ 2108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:25.194640 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:25.194643 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:25.195033 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:25.195221 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:25.197084 (+ 1863us) server_negotiation.cc:299] Negotiation successful
0806 02:55:25.197336 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14814,"thread_start_us":15448,"threads_started":1}
I20260806 02:55:25.207281 14123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62db4ddd40b64bb29919b6bb37253582: Bootstrap starting.
I20260806 02:55:25.193533 14250 master_runner.cc:186] Error getting master registration for 127.25.211.190:36995: OK, instance_id { permanent_uuid: "62db4ddd40b64bb29919b6bb37253582" instance_seqno: 1785984924014838 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 02:55:25.209010 14250 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260806 02:55:25.213109 14384 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d05d486fd0b84750b254692b1c6c362a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:55:25.213428 14384 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:55:25.214537 14123 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62db4ddd40b64bb29919b6bb37253582: Neither blocks nor log segments found. Creating new log.
I20260806 02:55:25.216070 14123 log.cc:824] T 00000000000000000000000000000000 P 62db4ddd40b64bb29919b6bb37253582: Log is configured to *not* fsync() on all Append() calls
I20260806 02:55:25.221748 14123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62db4ddd40b64bb29919b6bb37253582: No bootstrap required, opened a new log
I20260806 02:55:25.230710 14123 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62db4ddd40b64bb29919b6bb37253582 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62db4ddd40b64bb29919b6bb37253582" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 36995 } } peers { permanent_uuid: "95d94d857c6948dfa46874b44cc0bd75" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 33187 } } peers { permanent_uuid: "d05d486fd0b84750b254692b1c6c362a" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 43461 } }
I20260806 02:55:25.231094 14123 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62db4ddd40b64bb29919b6bb37253582 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:55:25.231212 14123 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62db4ddd40b64bb29919b6bb37253582 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62db4ddd40b64bb29919b6bb37253582, State: Initialized, Role: FOLLOWER
I20260806 02:55:25.231853 14123 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62db4ddd40b64bb29919b6bb37253582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62db4ddd40b64bb29919b6bb37253582" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 36995 } } peers { permanent_uuid: "95d94d857c6948dfa46874b44cc0bd75" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 33187 } } peers { permanent_uuid: "d05d486fd0b84750b254692b1c6c362a" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 43461 } }
I20260806 02:55:25.232859 14387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62db4ddd40b64bb29919b6bb37253582 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62db4ddd40b64bb29919b6bb37253582" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 36995 } } peers { permanent_uuid: "95d94d857c6948dfa46874b44cc0bd75" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 33187 } } peers { permanent_uuid: "d05d486fd0b84750b254692b1c6c362a" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 43461 } } }
I20260806 02:55:25.233304 14387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62db4ddd40b64bb29919b6bb37253582 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:55:25.234251 14123 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62db4ddd40b64bb29919b6bb37253582 [sys.catalog]: configured and running, proceeding with master startup.
W20260806 02:55:25.247113 14401 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 62db4ddd40b64bb29919b6bb37253582: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:55:25.247365 14401 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260806 02:55:25.254621 14051 tablet.cc:2369] T 00000000000000000000000000000000 P 62db4ddd40b64bb29919b6bb37253582: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:55:25.246984 14017 master_runner.cc:186] Error getting master registration for 127.25.211.189:33187: OK, instance_id { permanent_uuid: "95d94d857c6948dfa46874b44cc0bd75" instance_seqno: 1785984924539503 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 02:55:25.260213 14017 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 02:55:25.271023 14238 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.211.188" port: 43461 } attempt: 5
Aug 06 02:55:25 dist-test-slave-zj8q krb5kdc[13997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984924, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.189@KRBTEST.COM for kudu/127.25.211.188@KRBTEST.COM
I20260806 02:55:25.289924 14241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:25.271812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:43461 (local address 127.0.0.1:43148)
0806 02:55:25.272040 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:25.272063 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 02:55:25.272201 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:55:25.272612 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:55:25.272617 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:55:25.273174 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:55:25.273395 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:25.273407 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:25.274904 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:25.274910 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:25.276355 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:25.276372 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:25.277420 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:25.277426 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:25.277685 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:25.278425 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:55:25.278452 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:55:25.282384 (+ 3932us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:55:25.286833 (+ 4449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:25.286839 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:25.286844 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:25.287525 (+ 681us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:25.287987 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:25.287993 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:25.287997 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:25.288206 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:25.288862 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:55:25.288867 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:55:25.289029 (+ 162us) client_negotiation.cc:769] Sending connection context
0806 02:55:25.289193 (+ 164us) client_negotiation.cc:240] Negotiation successful
0806 02:55:25.289503 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260806 02:55:25.289989 14362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:25.272019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43148 (local address 127.25.211.188:43461)
0806 02:55:25.272226 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:25.272233 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:25.272256 (+ 23us) server_negotiation.cc:407] Connection header received
0806 02:55:25.272327 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:25.272336 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:25.272441 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:25.272567 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:25.273578 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:25.274760 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:25.276532 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:25.277253 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:25.277387 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:25.282612 (+ 5225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:25.282620 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:25.282625 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:25.282668 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:25.286639 (+ 3971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:25.287648 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:25.287655 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:25.287658 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:25.287870 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:25.288324 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:25.288329 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:25.288333 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:25.288738 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:25.288895 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:25.289323 (+ 428us) server_negotiation.cc:299] Negotiation successful
0806 02:55:25.289604 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260806 02:55:25.293833 14280 tablet.cc:2369] T 00000000000000000000000000000000 P d05d486fd0b84750b254692b1c6c362a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:55:25.296974 14238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95d94d857c6948dfa46874b44cc0bd75: Bootstrap starting.
I20260806 02:55:25.301555 14238 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95d94d857c6948dfa46874b44cc0bd75: Neither blocks nor log segments found. Creating new log.
I20260806 02:55:25.303195 14238 log.cc:824] T 00000000000000000000000000000000 P 95d94d857c6948dfa46874b44cc0bd75: Log is configured to *not* fsync() on all Append() calls
I20260806 02:55:25.306861 14238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95d94d857c6948dfa46874b44cc0bd75: No bootstrap required, opened a new log
I20260806 02:55:25.318090 14238 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95d94d857c6948dfa46874b44cc0bd75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62db4ddd40b64bb29919b6bb37253582" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 36995 } } peers { permanent_uuid: "95d94d857c6948dfa46874b44cc0bd75" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 33187 } } peers { permanent_uuid: "d05d486fd0b84750b254692b1c6c362a" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 43461 } }
I20260806 02:55:25.318576 14238 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95d94d857c6948dfa46874b44cc0bd75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:55:25.318702 14238 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95d94d857c6948dfa46874b44cc0bd75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95d94d857c6948dfa46874b44cc0bd75, State: Initialized, Role: FOLLOWER
I20260806 02:55:25.319723 14238 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95d94d857c6948dfa46874b44cc0bd75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62db4ddd40b64bb29919b6bb37253582" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 36995 } } peers { permanent_uuid: "95d94d857c6948dfa46874b44cc0bd75" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 33187 } } peers { permanent_uuid: "d05d486fd0b84750b254692b1c6c362a" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 43461 } }
I20260806 02:55:25.320180 14405 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95d94d857c6948dfa46874b44cc0bd75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:55:25.320505 14405 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95d94d857c6948dfa46874b44cc0bd75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62db4ddd40b64bb29919b6bb37253582" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 36995 } } peers { permanent_uuid: "95d94d857c6948dfa46874b44cc0bd75" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 33187 } } peers { permanent_uuid: "d05d486fd0b84750b254692b1c6c362a" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 43461 } }
I20260806 02:55:25.323252 14087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "95d94d857c6948dfa46874b44cc0bd75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62db4ddd40b64bb29919b6bb37253582" is_pre_election: true
I20260806 02:55:25.323956 14087 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 62db4ddd40b64bb29919b6bb37253582 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95d94d857c6948dfa46874b44cc0bd75 in term 0.
I20260806 02:55:25.324891 14163 leader_election.cc:302] T 00000000000000000000000000000000 P 95d94d857c6948dfa46874b44cc0bd75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62db4ddd40b64bb29919b6bb37253582, 95d94d857c6948dfa46874b44cc0bd75; no voters:
I20260806 02:55:25.326813 14406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95d94d857c6948dfa46874b44cc0bd75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62db4ddd40b64bb29919b6bb37253582" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 36995 } } peers { permanent_uuid: "95d94d857c6948dfa46874b44cc0bd75" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 33187 } } peers { permanent_uuid: "d05d486fd0b84750b254692b1c6c362a" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 43461 } } }
I20260806 02:55:25.329319 14406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95d94d857c6948dfa46874b44cc0bd75 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:55:25.329579 14406 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95d94d857c6948dfa46874b44cc0bd75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:55:25.329792 14406 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95d94d857c6948dfa46874b44cc0bd75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:55:25.329996 14406 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95d94d857c6948dfa46874b44cc0bd75 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:55:25.331921 14307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "95d94d857c6948dfa46874b44cc0bd75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d05d486fd0b84750b254692b1c6c362a" is_pre_election: true
I20260806 02:55:25.332755 14307 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d05d486fd0b84750b254692b1c6c362a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95d94d857c6948dfa46874b44cc0bd75 in term 0.
I20260806 02:55:25.333482 14406 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95d94d857c6948dfa46874b44cc0bd75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62db4ddd40b64bb29919b6bb37253582" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 36995 } } peers { permanent_uuid: "95d94d857c6948dfa46874b44cc0bd75" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 33187 } } peers { permanent_uuid: "d05d486fd0b84750b254692b1c6c362a" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 43461 } }
I20260806 02:55:25.329191 14238 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95d94d857c6948dfa46874b44cc0bd75 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:55:25.337057 14405 leader_election.cc:288] T 00000000000000000000000000000000 P 95d94d857c6948dfa46874b44cc0bd75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62db4ddd40b64bb29919b6bb37253582 (127.25.211.190:36995), d05d486fd0b84750b254692b1c6c362a (127.25.211.188:43461)
I20260806 02:55:25.337664 14406 leader_election.cc:288] T 00000000000000000000000000000000 P 95d94d857c6948dfa46874b44cc0bd75 [CANDIDATE]: Term 1 election: Requested vote from peers 62db4ddd40b64bb29919b6bb37253582 (127.25.211.190:36995), d05d486fd0b84750b254692b1c6c362a (127.25.211.188:43461)
I20260806 02:55:25.338119 14307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "95d94d857c6948dfa46874b44cc0bd75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d05d486fd0b84750b254692b1c6c362a"
I20260806 02:55:25.338408 14307 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d05d486fd0b84750b254692b1c6c362a [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:55:25.341933 14307 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d05d486fd0b84750b254692b1c6c362a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95d94d857c6948dfa46874b44cc0bd75 in term 1.
I20260806 02:55:25.342561 14161 leader_election.cc:302] T 00000000000000000000000000000000 P 95d94d857c6948dfa46874b44cc0bd75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95d94d857c6948dfa46874b44cc0bd75, d05d486fd0b84750b254692b1c6c362a; no voters:
I20260806 02:55:25.342927 14406 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95d94d857c6948dfa46874b44cc0bd75 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:55:25.343166 14406 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95d94d857c6948dfa46874b44cc0bd75 [term 1 LEADER]: Becoming Leader. State: Replica: 95d94d857c6948dfa46874b44cc0bd75, State: Running, Role: LEADER
I20260806 02:55:25.343883 14406 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95d94d857c6948dfa46874b44cc0bd75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62db4ddd40b64bb29919b6bb37253582" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 36995 } } peers { permanent_uuid: "95d94d857c6948dfa46874b44cc0bd75" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 33187 } } peers { permanent_uuid: "d05d486fd0b84750b254692b1c6c362a" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 43461 } }
I20260806 02:55:25.348243 14087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "95d94d857c6948dfa46874b44cc0bd75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62db4ddd40b64bb29919b6bb37253582"
I20260806 02:55:25.348507 14087 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62db4ddd40b64bb29919b6bb37253582 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:55:25.352080 14087 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 62db4ddd40b64bb29919b6bb37253582 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95d94d857c6948dfa46874b44cc0bd75 in term 1.
I20260806 02:55:25.359743 14405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95d94d857c6948dfa46874b44cc0bd75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95d94d857c6948dfa46874b44cc0bd75. Latest consensus state: current_term: 1 leader_uuid: "95d94d857c6948dfa46874b44cc0bd75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62db4ddd40b64bb29919b6bb37253582" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 36995 } } peers { permanent_uuid: "95d94d857c6948dfa46874b44cc0bd75" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 33187 } } peers { permanent_uuid: "d05d486fd0b84750b254692b1c6c362a" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 43461 } } }
I20260806 02:55:25.360302 14405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95d94d857c6948dfa46874b44cc0bd75 [sys.catalog]: This master's current role is: LEADER
I20260806 02:55:25.368494 14412 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:55:25.389524 14412 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:55:25.420594 14307 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d05d486fd0b84750b254692b1c6c362a [term 1 FOLLOWER]: Refusing update from remote peer 95d94d857c6948dfa46874b44cc0bd75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:55:25.421808 14405 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 95d94d857c6948dfa46874b44cc0bd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d05d486fd0b84750b254692b1c6c362a" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 43461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:55:25.429487 14087 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 62db4ddd40b64bb29919b6bb37253582 [term 1 FOLLOWER]: Refusing update from remote peer 95d94d857c6948dfa46874b44cc0bd75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:55:25.430677 14405 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 95d94d857c6948dfa46874b44cc0bd75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62db4ddd40b64bb29919b6bb37253582" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 36995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:55:25.471163 14387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62db4ddd40b64bb29919b6bb37253582 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95d94d857c6948dfa46874b44cc0bd75. Latest consensus state: current_term: 1 leader_uuid: "95d94d857c6948dfa46874b44cc0bd75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62db4ddd40b64bb29919b6bb37253582" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 36995 } } peers { permanent_uuid: "95d94d857c6948dfa46874b44cc0bd75" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 33187 } } peers { permanent_uuid: "d05d486fd0b84750b254692b1c6c362a" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 43461 } } }
I20260806 02:55:25.471539 14387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62db4ddd40b64bb29919b6bb37253582 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:55:25.475564 14419 mvcc.cc:205] Tried to move back new op lower bound from 7315394254490808320 to 7315394254215327744. Current Snapshot: MvccSnapshot[applied={T|T < 7315394254490808320}]
I20260806 02:55:25.475808 14424 mvcc.cc:205] Tried to move back new op lower bound from 7315394254490808320 to 7315394254215327744. Current Snapshot: MvccSnapshot[applied={T|T < 7315394254490808320}]
I20260806 02:55:25.479760 14364 sys_catalog.cc:455] T 00000000000000000000000000000000 P d05d486fd0b84750b254692b1c6c362a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95d94d857c6948dfa46874b44cc0bd75. Latest consensus state: current_term: 1 leader_uuid: "95d94d857c6948dfa46874b44cc0bd75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62db4ddd40b64bb29919b6bb37253582" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 36995 } } peers { permanent_uuid: "95d94d857c6948dfa46874b44cc0bd75" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 33187 } } peers { permanent_uuid: "d05d486fd0b84750b254692b1c6c362a" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 43461 } } }
I20260806 02:55:25.480106 14364 sys_catalog.cc:458] T 00000000000000000000000000000000 P d05d486fd0b84750b254692b1c6c362a [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:55:25.482615 14405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95d94d857c6948dfa46874b44cc0bd75 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "95d94d857c6948dfa46874b44cc0bd75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62db4ddd40b64bb29919b6bb37253582" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 36995 } } peers { permanent_uuid: "95d94d857c6948dfa46874b44cc0bd75" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 33187 } } peers { permanent_uuid: "d05d486fd0b84750b254692b1c6c362a" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 43461 } } }
I20260806 02:55:25.482940 14405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95d94d857c6948dfa46874b44cc0bd75 [sys.catalog]: This master's current role is: LEADER
I20260806 02:55:25.483320 14405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95d94d857c6948dfa46874b44cc0bd75 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "95d94d857c6948dfa46874b44cc0bd75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62db4ddd40b64bb29919b6bb37253582" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 36995 } } peers { permanent_uuid: "95d94d857c6948dfa46874b44cc0bd75" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 33187 } } peers { permanent_uuid: "d05d486fd0b84750b254692b1c6c362a" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 43461 } } }
I20260806 02:55:25.483577 14405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95d94d857c6948dfa46874b44cc0bd75 [sys.catalog]: This master's current role is: LEADER
I20260806 02:55:25.485024 14412 catalog_manager.cc:1347] Generated new cluster ID: b6545fe4726d489abe3e0b2f8b439b54
I20260806 02:55:25.485144 14412 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:55:25.486536 14422 mvcc.cc:205] Tried to move back new op lower bound from 7315394254490808320 to 7315394254215327744. Current Snapshot: MvccSnapshot[applied={T|T < 7315394254490808320}]
I20260806 02:55:25.487959 14387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62db4ddd40b64bb29919b6bb37253582 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "95d94d857c6948dfa46874b44cc0bd75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62db4ddd40b64bb29919b6bb37253582" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 36995 } } peers { permanent_uuid: "95d94d857c6948dfa46874b44cc0bd75" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 33187 } } peers { permanent_uuid: "d05d486fd0b84750b254692b1c6c362a" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 43461 } } }
I20260806 02:55:25.488185 14387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62db4ddd40b64bb29919b6bb37253582 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:55:25.489128 14364 sys_catalog.cc:455] T 00000000000000000000000000000000 P d05d486fd0b84750b254692b1c6c362a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "95d94d857c6948dfa46874b44cc0bd75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62db4ddd40b64bb29919b6bb37253582" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 36995 } } peers { permanent_uuid: "95d94d857c6948dfa46874b44cc0bd75" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 33187 } } peers { permanent_uuid: "d05d486fd0b84750b254692b1c6c362a" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 43461 } } }
I20260806 02:55:25.489398 14364 sys_catalog.cc:458] T 00000000000000000000000000000000 P d05d486fd0b84750b254692b1c6c362a [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:55:25.519618 14440 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 02:55:25.539412 14412 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:55:25.541405 14412 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:55:25.596508 14412 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95d94d857c6948dfa46874b44cc0bd75: Generated new TSK 0
I20260806 02:55:25.597802 14412 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:55:26.215971 14384 catalog_manager.cc:1259] Loaded cluster ID: b6545fe4726d489abe3e0b2f8b439b54
I20260806 02:55:26.216091 14384 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d05d486fd0b84750b254692b1c6c362a: loading cluster ID for follower catalog manager: success
I20260806 02:55:26.218787 14384 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d05d486fd0b84750b254692b1c6c362a: acquiring CA information for follower catalog manager: success
I20260806 02:55:26.220651 14384 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d05d486fd0b84750b254692b1c6c362a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 02:55:26.249485 14401 catalog_manager.cc:1259] Loaded cluster ID: b6545fe4726d489abe3e0b2f8b439b54
I20260806 02:55:26.249606 14401 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 62db4ddd40b64bb29919b6bb37253582: loading cluster ID for follower catalog manager: success
I20260806 02:55:26.252146 14401 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 62db4ddd40b64bb29919b6bb37253582: acquiring CA information for follower catalog manager: success
I20260806 02:55:26.253929 14401 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 62db4ddd40b64bb29919b6bb37253582: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 02:55:26.751744 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 14017
I20260806 02:55:26.753134 14118 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:55:26.863919 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 14017
I20260806 02:55:26.891963 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 14132
I20260806 02:55:26.893112 14232 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:55:27.001039 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 14132
I20260806 02:55:27.027479 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 14250
I20260806 02:55:27.028738 14339 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:55:27.137099 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 14250
2026-08-06T02:55:27Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (5081 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984707115702-26446-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:55:27 dist-test-slave-zj8q krb5kdc[14455](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:55:27 dist-test-slave-zj8q krb5kdc[14455](info): set up 2 sockets
Aug 06 02:55:27 dist-test-slave-zj8q krb5kdc[14455](info): commencing operation
krb5kdc: starting...
W20260806 02:55:28.378985 26446 mini_kdc.cc:121] Time spent starting KDC: real 1.194s user 0.000s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Aug 06 02:55:28 dist-test-slave-zj8q krb5kdc[14455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:55:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:55:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.211.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab.
Entry for principal kudu/127.25.211.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab.
I20260806 02:55:28.674014 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984707115702-26446-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984707115702-26446-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984707115702-26446-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:44563
--webserver_interface=127.25.211.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:36305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.211.190:44563
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:55:29.020272 14473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:55:29.020581 14473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:55:29.020705 14473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:55:29.028151 14473 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:55:29.028290 14473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:55:29.028342 14473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:55:29.028424 14473 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:55:29.028483 14473 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:55:29.039331 14473 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:36305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984707115702-26446-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984707115702-26446-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.211.190:44563
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:44563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.211.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984707115702-26446-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:55:29.040872 14473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:55:29.042392 14473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:55:29.055291 14480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:29.055636 14479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:29.055251 14484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:55:29.056062 14473 server_base.cc:1034] running on GCE node
I20260806 02:55:29.056929 14473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:55:29.058533 14473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:55:29.059767 14473 hybrid_clock.cc:648] HybridClock initialized: now 1785984929059711 us; error 45 us; skew 500 ppm
I20260806 02:55:29.060138 14473 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:55:29 dist-test-slave-zj8q krb5kdc[14455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984929, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:55:29.066660 14473 init.cc:377] Logged in from keytab as kudu/127.25.211.190@KRBTEST.COM (short username kudu)
I20260806 02:55:29.068318 14473 webserver.cc:460] Webserver started at http://127.25.211.190:36105/ using document root <none> and password file <none>
I20260806 02:55:29.069088 14473 fs_manager.cc:359] Metadata directory not provided
I20260806 02:55:29.069223 14473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:55:29.069548 14473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:55:29.072759 14473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984707115702-26446-0/minicluster-data/master-0/data/instance:
uuid: "28052007cbfe49229bfbdfd32791f714"
format_stamp: "Formatted at 2026-08-06 02:55:29 on dist-test-slave-zj8q"
server_key: "ae3bccf726ba63f961a9acf717486705"
server_key_iv: "391c230e814c1762c9478bbc6244a759"
server_key_version: "encryptionkey@0"
I20260806 02:55:29.073523 14473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984707115702-26446-0/minicluster-data/master-0/wal/instance:
uuid: "28052007cbfe49229bfbdfd32791f714"
format_stamp: "Formatted at 2026-08-06 02:55:29 on dist-test-slave-zj8q"
server_key: "ae3bccf726ba63f961a9acf717486705"
server_key_iv: "391c230e814c1762c9478bbc6244a759"
server_key_version: "encryptionkey@0"
I20260806 02:55:29.079092 14473 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260806 02:55:29.082979 14493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:29.084506 14473 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:55:29.084739 14473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984707115702-26446-0/minicluster-data/master-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984707115702-26446-0/minicluster-data/master-0/wal
uuid: "28052007cbfe49229bfbdfd32791f714"
format_stamp: "Formatted at 2026-08-06 02:55:29 on dist-test-slave-zj8q"
server_key: "ae3bccf726ba63f961a9acf717486705"
server_key_iv: "391c230e814c1762c9478bbc6244a759"
server_key_version: "encryptionkey@0"
I20260806 02:55:29.084954 14473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984707115702-26446-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984707115702-26446-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984707115702-26446-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:55:29.106300 14473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:55:29.111001 14473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:55:29.111303 14473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:55:29.129565 14473 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.190:44563
I20260806 02:55:29.129585 14573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.190:44563 every 8 connection(s)
I20260806 02:55:29.131042 14473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
I20260806 02:55:29.134626 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 14473
I20260806 02:55:29.134850 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1785984707115702-26446-0/minicluster-data/master-0/wal/instance
I20260806 02:55:29.135068 14574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:55:29.135561 26446 external_mini_cluster.cc:1428] Setting key 8411e6dd0c9049d34b8386dd3d624d2f
I20260806 02:55:29.149636 14574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28052007cbfe49229bfbdfd32791f714: Bootstrap starting.
Aug 06 02:55:29 dist-test-slave-zj8q krb5kdc[14455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:55:29.153658 14574 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28052007cbfe49229bfbdfd32791f714: Neither blocks nor log segments found. Creating new log.
I20260806 02:55:29.155210 14574 log.cc:824] T 00000000000000000000000000000000 P 28052007cbfe49229bfbdfd32791f714: Log is configured to *not* fsync() on all Append() calls
I20260806 02:55:29.158434 14574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28052007cbfe49229bfbdfd32791f714: No bootstrap required, opened a new log
I20260806 02:55:29.160774 14577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:29.137423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36028 (local address 127.25.211.190:44563)
0806 02:55:29.138349 (+ 926us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:29.138365 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:29.138420 (+ 55us) server_negotiation.cc:407] Connection header received
0806 02:55:29.140167 (+ 1747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:29.140200 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:29.140761 (+ 561us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:29.141244 (+ 483us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:29.142760 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:29.144712 (+ 1952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:29.146139 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:29.147273 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:29.147459 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:29.152051 (+ 4592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:29.152074 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:29.152095 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:29.152147 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:29.156361 (+ 4214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:29.157253 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:29.157261 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:29.157268 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:29.157452 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:29.157869 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:29.157873 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:29.157875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:29.158459 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:29.158743 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:29.159249 (+ 506us) server_negotiation.cc:299] Negotiation successful
0806 02:55:29.159710 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":365,"threads_started":1}
I20260806 02:55:29.164780 26446 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:55:29.166539 14574 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28052007cbfe49229bfbdfd32791f714 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28052007cbfe49229bfbdfd32791f714" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 44563 } }
I20260806 02:55:29.166976 14574 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28052007cbfe49229bfbdfd32791f714 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:55:29.167081 14574 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28052007cbfe49229bfbdfd32791f714 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28052007cbfe49229bfbdfd32791f714, State: Initialized, Role: FOLLOWER
I20260806 02:55:29.167662 14574 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28052007cbfe49229bfbdfd32791f714 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28052007cbfe49229bfbdfd32791f714" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 44563 } }
I20260806 02:55:29.167915 14574 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28052007cbfe49229bfbdfd32791f714 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:55:29.168057 14574 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28052007cbfe49229bfbdfd32791f714 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:55:29.168217 14574 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28052007cbfe49229bfbdfd32791f714 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:55:29.170490 14574 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28052007cbfe49229bfbdfd32791f714 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28052007cbfe49229bfbdfd32791f714" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 44563 } }
I20260806 02:55:29.171000 14574 leader_election.cc:302] T 00000000000000000000000000000000 P 28052007cbfe49229bfbdfd32791f714 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28052007cbfe49229bfbdfd32791f714; no voters:
I20260806 02:55:29.171604 14574 leader_election.cc:288] T 00000000000000000000000000000000 P 28052007cbfe49229bfbdfd32791f714 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:55:29.171749 14579 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28052007cbfe49229bfbdfd32791f714 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:55:29.172459 14579 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28052007cbfe49229bfbdfd32791f714 [term 1 LEADER]: Becoming Leader. State: Replica: 28052007cbfe49229bfbdfd32791f714, State: Running, Role: LEADER
I20260806 02:55:29.172999 14579 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28052007cbfe49229bfbdfd32791f714 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28052007cbfe49229bfbdfd32791f714" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 44563 } }
I20260806 02:55:29.173619 14574 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28052007cbfe49229bfbdfd32791f714 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:55:29.175896 14582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28052007cbfe49229bfbdfd32791f714 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28052007cbfe49229bfbdfd32791f714. Latest consensus state: current_term: 1 leader_uuid: "28052007cbfe49229bfbdfd32791f714" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28052007cbfe49229bfbdfd32791f714" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 44563 } } }
I20260806 02:55:29.176339 14582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28052007cbfe49229bfbdfd32791f714 [sys.catalog]: This master's current role is: LEADER
I20260806 02:55:29.176334 14581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28052007cbfe49229bfbdfd32791f714 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28052007cbfe49229bfbdfd32791f714" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28052007cbfe49229bfbdfd32791f714" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 44563 } } }
I20260806 02:55:29.176645 14581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28052007cbfe49229bfbdfd32791f714 [sys.catalog]: This master's current role is: LEADER
I20260806 02:55:29.177970 14591 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:55:29.183600 14591 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:55:29.193333 14591 catalog_manager.cc:1347] Generated new cluster ID: 0e32f47078d7461a97ca0cbc4eef5178
I20260806 02:55:29.193473 14591 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:55:29.203740 14591 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:55:29.205732 14591 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:55:29.219164 14591 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28052007cbfe49229bfbdfd32791f714: Generated new TSK 0
I20260806 02:55:29.220175 14591 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:55:29.278474 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 14473
I20260806 02:55:29.279948 14566 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:55:29.379801 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 14473
2026-08-06T02:55:29Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2239 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:55:29 dist-test-slave-zj8q krb5kdc[14605](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:55:29 dist-test-slave-zj8q krb5kdc[14605](info): set up 2 sockets
Aug 06 02:55:29 dist-test-slave-zj8q krb5kdc[14605](info): commencing operation
krb5kdc: starting...
W20260806 02:55:30.608249 26446 mini_kdc.cc:121] Time spent starting KDC: real 1.183s user 0.000s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Aug 06 02:55:30 dist-test-slave-zj8q krb5kdc[14605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:55:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:55:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.211.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab.
Entry for principal kudu/127.25.211.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab.
I20260806 02:55:30.917516 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:45343
--webserver_interface=127.25.211.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:36275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.211.190:45343
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:55:31.266521 14629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:55:31.266852 14629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:55:31.266980 14629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:55:31.274618 14629 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:55:31.274757 14629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:55:31.274806 14629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:55:31.274871 14629 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:55:31.274935 14629 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:55:31.285312 14629 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:36275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.211.190:45343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:45343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.211.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:55:31.286796 14629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:55:31.288192 14629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:55:31.300443 14638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:31.300599 14636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:31.300777 14635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:55:31.301678 14629 server_base.cc:1034] running on GCE node
I20260806 02:55:31.302582 14629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:55:31.304164 14629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:55:31.305405 14629 hybrid_clock.cc:648] HybridClock initialized: now 1785984931305349 us; error 45 us; skew 500 ppm
I20260806 02:55:31.305830 14629 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:55:31 dist-test-slave-zj8q krb5kdc[14605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984931, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:55:31.309085 14629 init.cc:377] Logged in from keytab as kudu/127.25.211.190@KRBTEST.COM (short username kudu)
I20260806 02:55:31.310748 14629 webserver.cc:460] Webserver started at http://127.25.211.190:34747/ using document root <none> and password file <none>
I20260806 02:55:31.311426 14629 fs_manager.cc:359] Metadata directory not provided
I20260806 02:55:31.311535 14629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:55:31.311816 14629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:55:31.315093 14629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/master-0/data/instance:
uuid: "7b628097688e47b7a0187a2f2f3a54e4"
format_stamp: "Formatted at 2026-08-06 02:55:31 on dist-test-slave-zj8q"
server_key: "4ef30fd60b93baa3a772901be83b1b44"
server_key_iv: "f0d33b683b62622baae7293d27b57725"
server_key_version: "encryptionkey@0"
I20260806 02:55:31.315884 14629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/master-0/wal/instance:
uuid: "7b628097688e47b7a0187a2f2f3a54e4"
format_stamp: "Formatted at 2026-08-06 02:55:31 on dist-test-slave-zj8q"
server_key: "4ef30fd60b93baa3a772901be83b1b44"
server_key_iv: "f0d33b683b62622baae7293d27b57725"
server_key_version: "encryptionkey@0"
I20260806 02:55:31.321669 14629 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260806 02:55:31.325510 14647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:31.327107 14629 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:55:31.327276 14629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/master-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/master-0/wal
uuid: "7b628097688e47b7a0187a2f2f3a54e4"
format_stamp: "Formatted at 2026-08-06 02:55:31 on dist-test-slave-zj8q"
server_key: "4ef30fd60b93baa3a772901be83b1b44"
server_key_iv: "f0d33b683b62622baae7293d27b57725"
server_key_version: "encryptionkey@0"
I20260806 02:55:31.327466 14629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:55:31.361478 14629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:55:31.365921 14629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:55:31.366197 14629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:55:31.385390 14629 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.190:45343
I20260806 02:55:31.385421 14726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.190:45343 every 8 connection(s)
I20260806 02:55:31.387405 14629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
I20260806 02:55:31.388383 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 14629
I20260806 02:55:31.388624 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/master-0/wal/instance
I20260806 02:55:31.389370 26446 external_mini_cluster.cc:1428] Setting key 64d925fc21b990898d58ba31c211316e
I20260806 02:55:31.391999 14728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:55:31.403086 14728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4: Bootstrap starting.
Aug 06 02:55:31 dist-test-slave-zj8q krb5kdc[14605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:55:31.406603 14728 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4: Neither blocks nor log segments found. Creating new log.
I20260806 02:55:31.407759 14728 log.cc:824] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4: Log is configured to *not* fsync() on all Append() calls
I20260806 02:55:31.411427 14728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4: No bootstrap required, opened a new log
I20260806 02:55:31.412670 14733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:31.391511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57680 (local address 127.25.211.190:45343)
0806 02:55:31.392419 (+ 908us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:31.392433 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:31.392476 (+ 43us) server_negotiation.cc:407] Connection header received
0806 02:55:31.394139 (+ 1663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:31.394205 (+ 66us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:31.394619 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:31.395004 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:31.396604 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:31.397974 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:31.399363 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:31.400095 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:31.400303 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:31.404618 (+ 4315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:31.404630 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:31.404648 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:31.404697 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:31.407823 (+ 3126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:31.408709 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:31.408716 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:31.408723 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:31.408910 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:31.409344 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:31.409349 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:31.409352 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:31.410123 (+ 771us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:31.410500 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:31.411074 (+ 574us) server_negotiation.cc:299] Negotiation successful
0806 02:55:31.411557 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":676,"thread_start_us":372,"threads_started":1}
I20260806 02:55:31.419857 14728 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b628097688e47b7a0187a2f2f3a54e4" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45343 } }
I20260806 02:55:31.420290 14728 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:55:31.420393 14728 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b628097688e47b7a0187a2f2f3a54e4, State: Initialized, Role: FOLLOWER
I20260806 02:55:31.420969 14728 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b628097688e47b7a0187a2f2f3a54e4" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45343 } }
I20260806 02:55:31.421372 14728 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:55:31.421542 14728 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:55:31.421762 14728 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:55:31.424116 14728 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b628097688e47b7a0187a2f2f3a54e4" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45343 } }
I20260806 02:55:31.424645 14728 leader_election.cc:302] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b628097688e47b7a0187a2f2f3a54e4; no voters:
I20260806 02:55:31.425280 14728 leader_election.cc:288] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:55:31.425431 14738 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:55:31.426229 14738 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4 [term 1 LEADER]: Becoming Leader. State: Replica: 7b628097688e47b7a0187a2f2f3a54e4, State: Running, Role: LEADER
I20260806 02:55:31.426738 14738 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b628097688e47b7a0187a2f2f3a54e4" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45343 } }
I20260806 02:55:31.427685 14728 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:55:31.429479 14739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b628097688e47b7a0187a2f2f3a54e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b628097688e47b7a0187a2f2f3a54e4" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45343 } } }
I20260806 02:55:31.429453 14740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b628097688e47b7a0187a2f2f3a54e4. Latest consensus state: current_term: 1 leader_uuid: "7b628097688e47b7a0187a2f2f3a54e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b628097688e47b7a0187a2f2f3a54e4" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45343 } } }
I20260806 02:55:31.429908 14740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4 [sys.catalog]: This master's current role is: LEADER
I20260806 02:55:31.429908 14739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4 [sys.catalog]: This master's current role is: LEADER
I20260806 02:55:31.431660 14744 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:55:31.437352 14744 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:55:31.444070 14765 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 02:55:31.447868 14744 catalog_manager.cc:1347] Generated new cluster ID: a5f8fb30977e47c9b2037092c1b5bb2d
I20260806 02:55:31.448004 14744 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:55:31.480521 14744 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:55:31.482017 14744 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:55:31.497247 14744 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4: Generated new TSK 0
I20260806 02:55:31.498322 14744 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.211.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab.
Entry for principal kudu/127.25.211.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab.
I20260806 02:55:31.577689 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.129:0
--local_ip_for_outbound_sockets=127.25.211.129
--webserver_interface=127.25.211.129
--webserver_port=0
--tserver_master_addrs=127.25.211.190:45343
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab
--principal=kudu/127.25.211.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:36275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:55:31.931034 14771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:55:31.931385 14771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:55:31.931486 14771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:55:31.939195 14771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:55:31.939318 14771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:55:31.939431 14771 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.129
I20260806 02:55:31.949515 14771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:36275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab
--principal=kudu/127.25.211.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.211.129
--webserver_port=0
--tserver_master_addrs=127.25.211.190:45343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.129
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:55:31.950857 14771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:55:31.952210 14771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:55:31.965566 14779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:31.965627 14785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:31.965600 14777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:55:31.971782 14771 server_base.cc:1034] running on GCE node
I20260806 02:55:31.972462 14771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:55:31.973450 14771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:55:31.974650 14771 hybrid_clock.cc:648] HybridClock initialized: now 1785984931974597 us; error 44 us; skew 500 ppm
I20260806 02:55:31.974973 14771 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:55:31 dist-test-slave-zj8q krb5kdc[14605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984931, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:55:31.978092 14771 init.cc:377] Logged in from keytab as kudu/127.25.211.129@KRBTEST.COM (short username kudu)
I20260806 02:55:31.979637 14771 webserver.cc:460] Webserver started at http://127.25.211.129:41871/ using document root <none> and password file <none>
I20260806 02:55:31.980319 14771 fs_manager.cc:359] Metadata directory not provided
I20260806 02:55:31.980418 14771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:55:31.980676 14771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:55:31.983644 14771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-0/data/instance:
uuid: "4c5c116a405d4e698eb9b15b3c50a17e"
format_stamp: "Formatted at 2026-08-06 02:55:31 on dist-test-slave-zj8q"
server_key: "3aa6a7f11d98841dc43f0e707e8bc873"
server_key_iv: "fce77d93f36ef050aebc4890ec034c96"
server_key_version: "encryptionkey@0"
I20260806 02:55:31.984383 14771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-0/wal/instance:
uuid: "4c5c116a405d4e698eb9b15b3c50a17e"
format_stamp: "Formatted at 2026-08-06 02:55:31 on dist-test-slave-zj8q"
server_key: "3aa6a7f11d98841dc43f0e707e8bc873"
server_key_iv: "fce77d93f36ef050aebc4890ec034c96"
server_key_version: "encryptionkey@0"
I20260806 02:55:31.989500 14771 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260806 02:55:31.992882 14793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:31.994230 14771 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:55:31.994387 14771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-0/wal
uuid: "4c5c116a405d4e698eb9b15b3c50a17e"
format_stamp: "Formatted at 2026-08-06 02:55:31 on dist-test-slave-zj8q"
server_key: "3aa6a7f11d98841dc43f0e707e8bc873"
server_key_iv: "fce77d93f36ef050aebc4890ec034c96"
server_key_version: "encryptionkey@0"
I20260806 02:55:31.994570 14771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:55:32.024163 14771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:55:32.028573 14771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:55:32.028888 14771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:55:32.029923 14771 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:55:32.031592 14771 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:55:32.031678 14771 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:32.031803 14771 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:55:32.031862 14771 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:32.062990 14771 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.129:43767
I20260806 02:55:32.063042 14964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.129:43767 every 8 connection(s)
I20260806 02:55:32.064321 14771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
I20260806 02:55:32.068424 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 14771
I20260806 02:55:32.068660 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-0/wal/instance
I20260806 02:55:32.069341 26446 external_mini_cluster.cc:1428] Setting key 108c8ddb37b2ae37ee15245a54a1e259
Aug 06 02:55:32 dist-test-slave-zj8q krb5kdc[14605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984931, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.129@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:55:32.085278 14967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:32.066934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:49777 (local address 127.25.211.190:45343)
0806 02:55:32.067498 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:32.067502 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:32.068502 (+ 1000us) server_negotiation.cc:407] Connection header received
0806 02:55:32.070581 (+ 2079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:32.070586 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:32.070654 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:32.070751 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:32.072644 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:32.073387 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:32.075136 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:32.075552 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:32.075683 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:32.079345 (+ 3662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:32.079352 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:32.079355 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:32.079395 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:32.081946 (+ 2551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:32.082819 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:32.082823 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:32.082825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:32.082961 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:32.083418 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:32.083421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:32.083423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:32.083729 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:32.083869 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:32.084654 (+ 785us) server_negotiation.cc:299] Negotiation successful
0806 02:55:32.084901 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":319,"threads_started":1}
I20260806 02:55:32.086385 14968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:32.067523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45343 (local address 127.25.211.129:49777)
0806 02:55:32.068318 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:32.068367 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 02:55:32.070313 (+ 1946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:55:32.071051 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:55:32.071075 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:55:32.071807 (+ 732us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:55:32.072447 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:32.072467 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:32.073575 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:32.073579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:32.074971 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:32.074986 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:32.075811 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:32.075819 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:32.076075 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:32.076747 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:55:32.076772 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:55:32.079085 (+ 2313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:55:32.082140 (+ 3055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:32.082171 (+ 31us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:32.082193 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:32.082680 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:32.083080 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:32.083086 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:32.083090 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:32.083295 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:32.083862 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:55:32.083871 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:55:32.084262 (+ 391us) client_negotiation.cc:769] Sending connection context
0806 02:55:32.084507 (+ 245us) client_negotiation.cc:240] Negotiation successful
0806 02:55:32.085092 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":338,"threads_started":1}
I20260806 02:55:32.088277 14965 heartbeater.cc:347] Connected to a master server at 127.25.211.190:45343
I20260806 02:55:32.088615 14965 heartbeater.cc:464] Registering TS with master...
I20260806 02:55:32.089425 14965 heartbeater.cc:511] Master 127.25.211.190:45343 requested a full tablet report, sending...
I20260806 02:55:32.092067 14677 ts_manager.cc:194] Registered new tserver with Master: 4c5c116a405d4e698eb9b15b3c50a17e (127.25.211.129:43767)
I20260806 02:55:32.094205 14677 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.211.129@KRBTEST.COM'} at 127.25.211.129:49777
WARNING: no policy specified for kudu/127.25.211.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab.
Entry for principal kudu/127.25.211.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab.
I20260806 02:55:32.137650 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.130:0
--local_ip_for_outbound_sockets=127.25.211.130
--webserver_interface=127.25.211.130
--webserver_port=0
--tserver_master_addrs=127.25.211.190:45343
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab
--principal=kudu/127.25.211.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:36275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:55:32.482265 14971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:55:32.482613 14971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:55:32.482717 14971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:55:32.490041 14971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:55:32.490196 14971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:55:32.490331 14971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.130
I20260806 02:55:32.500633 14971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:36275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab
--principal=kudu/127.25.211.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.211.130
--webserver_port=0
--tserver_master_addrs=127.25.211.190:45343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.130
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:55:32.501891 14971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:55:32.503252 14971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:55:32.516788 14987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:32.516867 14980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:32.516896 14984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:55:32.517452 14971 server_base.cc:1034] running on GCE node
I20260806 02:55:32.518326 14971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:55:32.519335 14971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:55:32.520543 14971 hybrid_clock.cc:648] HybridClock initialized: now 1785984932520487 us; error 54 us; skew 500 ppm
I20260806 02:55:32.520867 14971 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:55:32 dist-test-slave-zj8q krb5kdc[14605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984932, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:55:32.524135 14971 init.cc:377] Logged in from keytab as kudu/127.25.211.130@KRBTEST.COM (short username kudu)
I20260806 02:55:32.525736 14971 webserver.cc:460] Webserver started at http://127.25.211.130:37671/ using document root <none> and password file <none>
I20260806 02:55:32.526536 14971 fs_manager.cc:359] Metadata directory not provided
I20260806 02:55:32.526686 14971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:55:32.526998 14971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:55:32.530354 14971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-1/data/instance:
uuid: "0cb38587aca8444fb40d67e32941ea03"
format_stamp: "Formatted at 2026-08-06 02:55:32 on dist-test-slave-zj8q"
server_key: "98e38b2e4a4e0b25bca5309f3b32bde0"
server_key_iv: "1384851389e32776e8555760b7c71c14"
server_key_version: "encryptionkey@0"
I20260806 02:55:32.531291 14971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-1/wal/instance:
uuid: "0cb38587aca8444fb40d67e32941ea03"
format_stamp: "Formatted at 2026-08-06 02:55:32 on dist-test-slave-zj8q"
server_key: "98e38b2e4a4e0b25bca5309f3b32bde0"
server_key_iv: "1384851389e32776e8555760b7c71c14"
server_key_version: "encryptionkey@0"
I20260806 02:55:32.537144 14971 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260806 02:55:32.540846 14997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:32.542234 14971 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:55:32.542400 14971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-1/wal
uuid: "0cb38587aca8444fb40d67e32941ea03"
format_stamp: "Formatted at 2026-08-06 02:55:32 on dist-test-slave-zj8q"
server_key: "98e38b2e4a4e0b25bca5309f3b32bde0"
server_key_iv: "1384851389e32776e8555760b7c71c14"
server_key_version: "encryptionkey@0"
I20260806 02:55:32.542603 14971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:55:32.573500 14971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:55:32.578003 14971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:55:32.578354 14971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:55:32.579407 14971 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:55:32.581096 14971 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:55:32.581198 14971 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:32.581347 14971 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:55:32.581411 14971 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:32.612905 14971 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.130:32867
I20260806 02:55:32.612962 15179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.130:32867 every 8 connection(s)
I20260806 02:55:32.614460 14971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
I20260806 02:55:32.618750 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 14971
I20260806 02:55:32.618942 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-1/wal/instance
I20260806 02:55:32.619390 26446 external_mini_cluster.cc:1428] Setting key b2c9a1046064210f968f1ab5111897ca
Aug 06 02:55:32 dist-test-slave-zj8q krb5kdc[14605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984932, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.130@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:55:32.636538 15182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:32.617084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:54881 (local address 127.25.211.190:45343)
0806 02:55:32.617524 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:32.617528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:32.618478 (+ 950us) server_negotiation.cc:407] Connection header received
0806 02:55:32.620495 (+ 2017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:32.620501 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:32.620585 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:32.620684 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:32.622464 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:32.623227 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:32.624958 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:32.625542 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:32.625680 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:32.629867 (+ 4187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:32.629874 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:32.629879 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:32.629925 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:32.633183 (+ 3258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:32.634047 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:32.634051 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:32.634053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:32.634247 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:32.634731 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:32.634736 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:32.634740 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:32.635025 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:32.635175 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:32.635905 (+ 730us) server_negotiation.cc:299] Negotiation successful
0806 02:55:32.636145 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":234,"threads_started":1}
I20260806 02:55:32.637490 15184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:32.617537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45343 (local address 127.25.211.130:54881)
0806 02:55:32.618313 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:32.618365 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 02:55:32.620217 (+ 1852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:55:32.620970 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:55:32.620979 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:55:32.621658 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:55:32.622297 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:32.622316 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:32.623383 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:32.623387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:32.624793 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:32.624807 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:32.625755 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:32.625760 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:32.626004 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:32.626732 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:55:32.626765 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:55:32.629549 (+ 2784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:55:32.633390 (+ 3841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:32.633399 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:32.633414 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:32.633890 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:32.634384 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:32.634389 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:32.634393 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:32.634605 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:32.635154 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:55:32.635163 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:55:32.635525 (+ 362us) client_negotiation.cc:769] Sending connection context
0806 02:55:32.635785 (+ 260us) client_negotiation.cc:240] Negotiation successful
0806 02:55:32.636347 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":296,"threads_started":1}
I20260806 02:55:32.638970 15180 heartbeater.cc:347] Connected to a master server at 127.25.211.190:45343
I20260806 02:55:32.639279 15180 heartbeater.cc:464] Registering TS with master...
I20260806 02:55:32.640012 15180 heartbeater.cc:511] Master 127.25.211.190:45343 requested a full tablet report, sending...
I20260806 02:55:32.641521 14677 ts_manager.cc:194] Registered new tserver with Master: 0cb38587aca8444fb40d67e32941ea03 (127.25.211.130:32867)
I20260806 02:55:32.642665 14677 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.211.130@KRBTEST.COM'} at 127.25.211.130:54881
WARNING: no policy specified for kudu/127.25.211.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab.
Entry for principal kudu/127.25.211.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab.
I20260806 02:55:32.685132 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.131:0
--local_ip_for_outbound_sockets=127.25.211.131
--webserver_interface=127.25.211.131
--webserver_port=0
--tserver_master_addrs=127.25.211.190:45343
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab
--principal=kudu/127.25.211.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:36275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:55:33.025457 15188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:55:33.025780 15188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:55:33.025877 15188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:55:33.033771 15188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:55:33.033905 15188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:55:33.034058 15188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.131
I20260806 02:55:33.044224 15188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:36275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab
--principal=kudu/127.25.211.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.211.131
--webserver_port=0
--tserver_master_addrs=127.25.211.190:45343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.131
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:55:33.045603 15188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:55:33.047013 15188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:55:33.060119 15196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:33.060148 15197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:33.060189 15200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:55:33.061002 15188 server_base.cc:1034] running on GCE node
I20260806 02:55:33.061684 15188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:55:33.062783 15188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:55:33.064020 15188 hybrid_clock.cc:648] HybridClock initialized: now 1785984933063956 us; error 54 us; skew 500 ppm
I20260806 02:55:33.064368 15188 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:55:33 dist-test-slave-zj8q krb5kdc[14605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984933, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:55:33.067795 15188 init.cc:377] Logged in from keytab as kudu/127.25.211.131@KRBTEST.COM (short username kudu)
I20260806 02:55:33.069285 15188 webserver.cc:460] Webserver started at http://127.25.211.131:42341/ using document root <none> and password file <none>
I20260806 02:55:33.069983 15188 fs_manager.cc:359] Metadata directory not provided
I20260806 02:55:33.070101 15188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:55:33.070432 15188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:55:33.073372 15188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-2/data/instance:
uuid: "7acfb5f0395a47608eac1775aa86e11b"
format_stamp: "Formatted at 2026-08-06 02:55:33 on dist-test-slave-zj8q"
server_key: "9aa9fb40458463c7dd8c14404dab453e"
server_key_iv: "cb04e582455cef0709d5070676529619"
server_key_version: "encryptionkey@0"
I20260806 02:55:33.074236 15188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-2/wal/instance:
uuid: "7acfb5f0395a47608eac1775aa86e11b"
format_stamp: "Formatted at 2026-08-06 02:55:33 on dist-test-slave-zj8q"
server_key: "9aa9fb40458463c7dd8c14404dab453e"
server_key_iv: "cb04e582455cef0709d5070676529619"
server_key_version: "encryptionkey@0"
I20260806 02:55:33.079982 15188 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 02:55:33.083738 15211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:33.085165 15188 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 02:55:33.085338 15188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-2/wal
uuid: "7acfb5f0395a47608eac1775aa86e11b"
format_stamp: "Formatted at 2026-08-06 02:55:33 on dist-test-slave-zj8q"
server_key: "9aa9fb40458463c7dd8c14404dab453e"
server_key_iv: "cb04e582455cef0709d5070676529619"
server_key_version: "encryptionkey@0"
I20260806 02:55:33.085549 15188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:55:33.097736 14965 heartbeater.cc:503] Master 127.25.211.190:45343 was elected leader, sending a full tablet report...
I20260806 02:55:33.116549 15188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:55:33.120995 15188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:55:33.121287 15188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:55:33.122409 15188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:55:33.123896 15188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:55:33.123978 15188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:33.124099 15188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:55:33.124158 15188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:33.155496 15188 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.131:39655
I20260806 02:55:33.155539 15384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.131:39655 every 8 connection(s)
I20260806 02:55:33.156807 15188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
I20260806 02:55:33.165227 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 15188
I20260806 02:55:33.165441 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-2/wal/instance
I20260806 02:55:33.165926 26446 external_mini_cluster.cc:1428] Setting key b083d16a6fae49edf7a63e6a67816f14
Aug 06 02:55:33 dist-test-slave-zj8q krb5kdc[14605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984933, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.131@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:55:33.176957 15387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:33.159343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:33991 (local address 127.25.211.190:45343)
0806 02:55:33.159838 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:33.159844 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:33.160797 (+ 953us) server_negotiation.cc:407] Connection header received
0806 02:55:33.162664 (+ 1867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:33.162669 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:33.162758 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:33.162857 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:33.164391 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:33.165345 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:33.167058 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:33.167447 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:33.167573 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:33.171243 (+ 3670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:33.171248 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:33.171251 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:33.171287 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:33.173752 (+ 2465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:33.174575 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:33.174580 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:33.174582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:33.174727 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:33.175155 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:33.175159 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:33.175163 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:33.175444 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:33.175593 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:33.176305 (+ 712us) server_negotiation.cc:299] Negotiation successful
0806 02:55:33.176548 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":199,"threads_started":1}
I20260806 02:55:33.177927 15389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:33.159915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45343 (local address 127.25.211.131:33991)
0806 02:55:33.160630 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:33.160675 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 02:55:33.162385 (+ 1710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:55:33.163095 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:55:33.163104 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:55:33.163641 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:55:33.164227 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:33.164244 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:33.165512 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:33.165529 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:33.166904 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:33.166917 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:33.167674 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:33.167681 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:33.167921 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:33.168646 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:55:33.168676 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:55:33.171013 (+ 2337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:55:33.173924 (+ 2911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:33.173932 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:33.173946 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:33.174448 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:33.174836 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:33.174840 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:33.174843 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:33.175039 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:33.175576 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:55:33.175584 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:55:33.175940 (+ 356us) client_negotiation.cc:769] Sending connection context
0806 02:55:33.176189 (+ 249us) client_negotiation.cc:240] Negotiation successful
0806 02:55:33.176754 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":285,"threads_started":1}
I20260806 02:55:33.179349 15385 heartbeater.cc:347] Connected to a master server at 127.25.211.190:45343
I20260806 02:55:33.179716 15385 heartbeater.cc:464] Registering TS with master...
I20260806 02:55:33.180619 15385 heartbeater.cc:511] Master 127.25.211.190:45343 requested a full tablet report, sending...
I20260806 02:55:33.182188 14677 ts_manager.cc:194] Registered new tserver with Master: 7acfb5f0395a47608eac1775aa86e11b (127.25.211.131:39655)
I20260806 02:55:33.183089 14677 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.211.131@KRBTEST.COM'} at 127.25.211.131:33991
I20260806 02:55:33.193843 26446 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:55:33.211367 15387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:33.198558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43076 (local address 127.25.211.190:45343)
0806 02:55:33.198727 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:33.198731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:33.199189 (+ 458us) server_negotiation.cc:407] Connection header received
0806 02:55:33.199366 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:33.199370 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:33.199470 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:33.199557 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:33.200926 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:33.201678 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:33.203110 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:33.203556 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:33.203660 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:33.206219 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:33.206226 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:33.206230 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:33.206269 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:33.208685 (+ 2416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:33.209378 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:33.209382 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:33.209384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:33.209518 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:33.209956 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:33.209960 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:33.209962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:33.210373 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:33.210566 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:33.210940 (+ 374us) server_negotiation.cc:299] Negotiation successful
0806 02:55:33.211136 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 02:55:33.215859 14677 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43076:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:55:33.222909 14677 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:55:33.260707 15408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:33.236328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53768 (local address 127.25.211.131:39655)
0806 02:55:33.250750 (+ 14422us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:33.250761 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:33.250807 (+ 46us) server_negotiation.cc:407] Connection header received
0806 02:55:33.250943 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:33.250953 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:33.251283 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:33.251555 (+ 272us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:55:33.253445 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:33.255056 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:33.257960 (+ 2904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:33.259473 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:33.259663 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:33.259864 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:33.260126 (+ 262us) server_negotiation.cc:299] Negotiation successful
0806 02:55:33.260305 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14236,"thread_start_us":14072,"threads_started":1}
I20260806 02:55:33.262179 15409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:33.237035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.130:32867 (local address 127.0.0.1:35904)
0806 02:55:33.252433 (+ 15398us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:33.252456 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 02:55:33.252594 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:55:33.253820 (+ 1226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:55:33.253825 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:55:33.253854 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:55:33.255516 (+ 1662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:33.255530 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:33.257711 (+ 2181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:33.257720 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:33.259242 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:33.259256 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:33.261123 (+ 1867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:33.261130 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:33.261300 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:33.261334 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 02:55:33.261405 (+ 71us) client_negotiation.cc:240] Negotiation successful
0806 02:55:33.261611 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15258,"mutex_wait_us":68,"thread_start_us":16972,"threads_started":1}
I20260806 02:55:33.262321 15410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:33.237347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35904 (local address 127.25.211.130:32867)
0806 02:55:33.253251 (+ 15904us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:33.253261 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:33.253292 (+ 31us) server_negotiation.cc:407] Connection header received
0806 02:55:33.253400 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:33.253407 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:33.253607 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:33.253814 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:55:33.256016 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:33.257525 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:33.259468 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:33.260875 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:33.261542 (+ 667us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:33.261717 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:33.261814 (+ 97us) server_negotiation.cc:299] Negotiation successful
0806 02:55:33.261975 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15692,"thread_start_us":15807,"threads_started":1}
I20260806 02:55:33.262365 15403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:33.235778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:43767 (local address 127.0.0.1:60098)
0806 02:55:33.236492 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:33.236541 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 02:55:33.236714 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:55:33.254018 (+ 17304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:55:33.254025 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:55:33.254053 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:55:33.255444 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:33.255464 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:33.257183 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:33.257187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:33.259182 (+ 1995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:33.259199 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:33.261439 (+ 2240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:33.261445 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:33.261622 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:33.261651 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 02:55:33.261736 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 02:55:33.261878 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"mutex_wait_us":22,"thread_start_us":347,"threads_started":1}
I20260806 02:55:33.262200 15404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:33.235806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.131:39655 (local address 127.0.0.1:53768)
0806 02:55:33.247882 (+ 12076us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:33.247912 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0806 02:55:33.248116 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:55:33.251939 (+ 3823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:55:33.251954 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:55:33.251997 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:55:33.252615 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:33.252630 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:33.255209 (+ 2579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:33.255216 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:33.257777 (+ 2561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:33.257789 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:33.259663 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:33.259670 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:33.259869 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:33.259924 (+ 55us) client_negotiation.cc:769] Sending connection context
0806 02:55:33.261337 (+ 1413us) client_negotiation.cc:240] Negotiation successful
0806 02:55:33.261619 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11878,"mutex_wait_us":2,"thread_start_us":12436,"threads_started":1}
I20260806 02:55:33.263438 15406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:33.235904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60098 (local address 127.25.211.129:43767)
0806 02:55:33.253398 (+ 17494us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:33.253406 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:33.253431 (+ 25us) server_negotiation.cc:407] Connection header received
0806 02:55:33.253523 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:33.253531 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:33.253785 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:33.254188 (+ 403us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:55:33.255640 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:33.257047 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:33.259369 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:33.260879 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:33.261080 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:33.261283 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:33.262936 (+ 1653us) server_negotiation.cc:299] Negotiation successful
0806 02:55:33.263113 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17329,"thread_start_us":17184,"threads_started":1}
I20260806 02:55:33.265978 15077 tablet_service.cc:1467] Processing CreateTablet for tablet 51456b87205647a7ae576e09acbe7017 (DEFAULT_TABLE table=test-table [id=ff4ff664d41b4f3b9d2cb0bbbee2f3e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:55:33.265981 15290 tablet_service.cc:1467] Processing CreateTablet for tablet 51456b87205647a7ae576e09acbe7017 (DEFAULT_TABLE table=test-table [id=ff4ff664d41b4f3b9d2cb0bbbee2f3e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:55:33.267465 15290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51456b87205647a7ae576e09acbe7017. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:55:33.267499 15077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51456b87205647a7ae576e09acbe7017. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:55:33.269817 14862 tablet_service.cc:1467] Processing CreateTablet for tablet 51456b87205647a7ae576e09acbe7017 (DEFAULT_TABLE table=test-table [id=ff4ff664d41b4f3b9d2cb0bbbee2f3e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:55:33.271345 14862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51456b87205647a7ae576e09acbe7017. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:55:33.278759 15411 tablet_bootstrap.cc:492] T 51456b87205647a7ae576e09acbe7017 P 7acfb5f0395a47608eac1775aa86e11b: Bootstrap starting.
I20260806 02:55:33.282641 15411 tablet_bootstrap.cc:654] T 51456b87205647a7ae576e09acbe7017 P 7acfb5f0395a47608eac1775aa86e11b: Neither blocks nor log segments found. Creating new log.
I20260806 02:55:33.282946 15412 tablet_bootstrap.cc:492] T 51456b87205647a7ae576e09acbe7017 P 0cb38587aca8444fb40d67e32941ea03: Bootstrap starting.
I20260806 02:55:33.284103 15411 log.cc:824] T 51456b87205647a7ae576e09acbe7017 P 7acfb5f0395a47608eac1775aa86e11b: Log is configured to *not* fsync() on all Append() calls
I20260806 02:55:33.286466 15412 tablet_bootstrap.cc:654] T 51456b87205647a7ae576e09acbe7017 P 0cb38587aca8444fb40d67e32941ea03: Neither blocks nor log segments found. Creating new log.
I20260806 02:55:33.287405 15411 tablet_bootstrap.cc:492] T 51456b87205647a7ae576e09acbe7017 P 7acfb5f0395a47608eac1775aa86e11b: No bootstrap required, opened a new log
I20260806 02:55:33.287701 15411 ts_tablet_manager.cc:1397] T 51456b87205647a7ae576e09acbe7017 P 7acfb5f0395a47608eac1775aa86e11b: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260806 02:55:33.287710 15412 log.cc:824] T 51456b87205647a7ae576e09acbe7017 P 0cb38587aca8444fb40d67e32941ea03: Log is configured to *not* fsync() on all Append() calls
I20260806 02:55:33.288184 15416 tablet_bootstrap.cc:492] T 51456b87205647a7ae576e09acbe7017 P 4c5c116a405d4e698eb9b15b3c50a17e: Bootstrap starting.
I20260806 02:55:33.291620 15416 tablet_bootstrap.cc:654] T 51456b87205647a7ae576e09acbe7017 P 4c5c116a405d4e698eb9b15b3c50a17e: Neither blocks nor log segments found. Creating new log.
I20260806 02:55:33.292588 15416 log.cc:824] T 51456b87205647a7ae576e09acbe7017 P 4c5c116a405d4e698eb9b15b3c50a17e: Log is configured to *not* fsync() on all Append() calls
I20260806 02:55:33.292953 15412 tablet_bootstrap.cc:492] T 51456b87205647a7ae576e09acbe7017 P 0cb38587aca8444fb40d67e32941ea03: No bootstrap required, opened a new log
I20260806 02:55:33.293298 15412 ts_tablet_manager.cc:1397] T 51456b87205647a7ae576e09acbe7017 P 0cb38587aca8444fb40d67e32941ea03: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260806 02:55:33.294960 15416 tablet_bootstrap.cc:492] T 51456b87205647a7ae576e09acbe7017 P 4c5c116a405d4e698eb9b15b3c50a17e: No bootstrap required, opened a new log
I20260806 02:55:33.295244 15416 ts_tablet_manager.cc:1397] T 51456b87205647a7ae576e09acbe7017 P 4c5c116a405d4e698eb9b15b3c50a17e: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20260806 02:55:33.298579 15411 raft_consensus.cc:348] T 51456b87205647a7ae576e09acbe7017 P 7acfb5f0395a47608eac1775aa86e11b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c5c116a405d4e698eb9b15b3c50a17e" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 43767 } } peers { permanent_uuid: "7acfb5f0395a47608eac1775aa86e11b" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 39655 } } peers { permanent_uuid: "0cb38587aca8444fb40d67e32941ea03" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 32867 } }
I20260806 02:55:33.299213 15411 raft_consensus.cc:374] T 51456b87205647a7ae576e09acbe7017 P 7acfb5f0395a47608eac1775aa86e11b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:55:33.299360 15411 raft_consensus.cc:729] T 51456b87205647a7ae576e09acbe7017 P 7acfb5f0395a47608eac1775aa86e11b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7acfb5f0395a47608eac1775aa86e11b, State: Initialized, Role: FOLLOWER
I20260806 02:55:33.300216 15411 consensus_queue.cc:260] T 51456b87205647a7ae576e09acbe7017 P 7acfb5f0395a47608eac1775aa86e11b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c5c116a405d4e698eb9b15b3c50a17e" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 43767 } } peers { permanent_uuid: "7acfb5f0395a47608eac1775aa86e11b" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 39655 } } peers { permanent_uuid: "0cb38587aca8444fb40d67e32941ea03" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 32867 } }
I20260806 02:55:33.303007 15416 raft_consensus.cc:348] T 51456b87205647a7ae576e09acbe7017 P 4c5c116a405d4e698eb9b15b3c50a17e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c5c116a405d4e698eb9b15b3c50a17e" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 43767 } } peers { permanent_uuid: "7acfb5f0395a47608eac1775aa86e11b" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 39655 } } peers { permanent_uuid: "0cb38587aca8444fb40d67e32941ea03" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 32867 } }
I20260806 02:55:33.303406 15416 raft_consensus.cc:374] T 51456b87205647a7ae576e09acbe7017 P 4c5c116a405d4e698eb9b15b3c50a17e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:55:33.303556 15416 raft_consensus.cc:729] T 51456b87205647a7ae576e09acbe7017 P 4c5c116a405d4e698eb9b15b3c50a17e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c5c116a405d4e698eb9b15b3c50a17e, State: Initialized, Role: FOLLOWER
I20260806 02:55:33.304222 15385 heartbeater.cc:503] Master 127.25.211.190:45343 was elected leader, sending a full tablet report...
I20260806 02:55:33.304080 15412 raft_consensus.cc:348] T 51456b87205647a7ae576e09acbe7017 P 0cb38587aca8444fb40d67e32941ea03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c5c116a405d4e698eb9b15b3c50a17e" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 43767 } } peers { permanent_uuid: "7acfb5f0395a47608eac1775aa86e11b" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 39655 } } peers { permanent_uuid: "0cb38587aca8444fb40d67e32941ea03" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 32867 } }
I20260806 02:55:33.304322 15416 consensus_queue.cc:260] T 51456b87205647a7ae576e09acbe7017 P 4c5c116a405d4e698eb9b15b3c50a17e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c5c116a405d4e698eb9b15b3c50a17e" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 43767 } } peers { permanent_uuid: "7acfb5f0395a47608eac1775aa86e11b" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 39655 } } peers { permanent_uuid: "0cb38587aca8444fb40d67e32941ea03" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 32867 } }
I20260806 02:55:33.304567 15412 raft_consensus.cc:374] T 51456b87205647a7ae576e09acbe7017 P 0cb38587aca8444fb40d67e32941ea03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:55:33.304682 15412 raft_consensus.cc:729] T 51456b87205647a7ae576e09acbe7017 P 0cb38587aca8444fb40d67e32941ea03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cb38587aca8444fb40d67e32941ea03, State: Initialized, Role: FOLLOWER
I20260806 02:55:33.305296 15412 consensus_queue.cc:260] T 51456b87205647a7ae576e09acbe7017 P 0cb38587aca8444fb40d67e32941ea03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c5c116a405d4e698eb9b15b3c50a17e" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 43767 } } peers { permanent_uuid: "7acfb5f0395a47608eac1775aa86e11b" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 39655 } } peers { permanent_uuid: "0cb38587aca8444fb40d67e32941ea03" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 32867 } }
I20260806 02:55:33.306927 15411 ts_tablet_manager.cc:1428] T 51456b87205647a7ae576e09acbe7017 P 7acfb5f0395a47608eac1775aa86e11b: Time spent starting tablet: real 0.019s user 0.009s sys 0.006s
I20260806 02:55:33.308017 15180 heartbeater.cc:503] Master 127.25.211.190:45343 was elected leader, sending a full tablet report...
I20260806 02:55:33.309890 15428 raft_consensus.cc:482] T 51456b87205647a7ae576e09acbe7017 P 0cb38587aca8444fb40d67e32941ea03 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:55:33.310207 15428 raft_consensus.cc:504] T 51456b87205647a7ae576e09acbe7017 P 0cb38587aca8444fb40d67e32941ea03 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c5c116a405d4e698eb9b15b3c50a17e" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 43767 } } peers { permanent_uuid: "7acfb5f0395a47608eac1775aa86e11b" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 39655 } } peers { permanent_uuid: "0cb38587aca8444fb40d67e32941ea03" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 32867 } }
I20260806 02:55:33.310169 15412 ts_tablet_manager.cc:1428] T 51456b87205647a7ae576e09acbe7017 P 0cb38587aca8444fb40d67e32941ea03: Time spent starting tablet: real 0.017s user 0.015s sys 0.000s
I20260806 02:55:33.312662 15422 raft_consensus.cc:482] T 51456b87205647a7ae576e09acbe7017 P 4c5c116a405d4e698eb9b15b3c50a17e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:55:33.312232 15428 leader_election.cc:288] T 51456b87205647a7ae576e09acbe7017 P 0cb38587aca8444fb40d67e32941ea03 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c5c116a405d4e698eb9b15b3c50a17e (127.25.211.129:43767), 7acfb5f0395a47608eac1775aa86e11b (127.25.211.131:39655)
I20260806 02:55:33.313045 15422 raft_consensus.cc:504] T 51456b87205647a7ae576e09acbe7017 P 4c5c116a405d4e698eb9b15b3c50a17e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c5c116a405d4e698eb9b15b3c50a17e" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 43767 } } peers { permanent_uuid: "7acfb5f0395a47608eac1775aa86e11b" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 39655 } } peers { permanent_uuid: "0cb38587aca8444fb40d67e32941ea03" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 32867 } }
I20260806 02:55:33.315208 15422 leader_election.cc:288] T 51456b87205647a7ae576e09acbe7017 P 4c5c116a405d4e698eb9b15b3c50a17e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7acfb5f0395a47608eac1775aa86e11b (127.25.211.131:39655), 0cb38587aca8444fb40d67e32941ea03 (127.25.211.130:32867)
I20260806 02:55:33.315702 15416 ts_tablet_manager.cc:1428] T 51456b87205647a7ae576e09acbe7017 P 4c5c116a405d4e698eb9b15b3c50a17e: Time spent starting tablet: real 0.020s user 0.009s sys 0.004s
W20260806 02:55:33.316713 14966 tablet.cc:2369] T 51456b87205647a7ae576e09acbe7017 P 4c5c116a405d4e698eb9b15b3c50a17e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:55:33.325210 15408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:33.312993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:56713 (local address 127.25.211.131:39655)
0806 02:55:33.313220 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:33.313225 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:33.317896 (+ 4671us) server_negotiation.cc:407] Connection header received
0806 02:55:33.318132 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:33.318137 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:33.318249 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:33.318332 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:55:33.319119 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:33.320202 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:33.322741 (+ 2539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:33.323925 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:33.324065 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:33.324144 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:33.324780 (+ 636us) server_negotiation.cc:299] Negotiation successful
0806 02:55:33.324920 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260806 02:55:33.325359 15406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:33.312956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:35605 (local address 127.25.211.129:43767)
0806 02:55:33.313179 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:33.313189 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:33.317576 (+ 4387us) server_negotiation.cc:407] Connection header received
0806 02:55:33.317656 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:33.317661 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:33.317736 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:33.317825 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:55:33.318577 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:33.319657 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:33.322531 (+ 2874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:33.324118 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:33.324371 (+ 253us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:33.324449 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:33.324985 (+ 536us) server_negotiation.cc:299] Negotiation successful
0806 02:55:33.325108 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":69}
I20260806 02:55:33.325872 15433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:33.312524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:43767 (local address 127.25.211.130:35605)
0806 02:55:33.317076 (+ 4552us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:33.317098 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 02:55:33.317257 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:55:33.317984 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:55:33.317988 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:55:33.318016 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:55:33.318400 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:33.318412 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:33.319846 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:33.319850 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:33.322321 (+ 2471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:33.322339 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:33.324641 (+ 2302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:33.324646 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:33.324806 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:33.324840 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 02:55:33.324918 (+ 78us) client_negotiation.cc:240] Negotiation successful
0806 02:55:33.325057 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4381,"thread_start_us":4973,"threads_started":1}
I20260806 02:55:33.326058 15430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:33.313756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.131:39655 (local address 127.25.211.130:56713)
0806 02:55:33.317803 (+ 4047us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:33.317819 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:55:33.317985 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:55:33.318501 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:55:33.318506 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:55:33.318531 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:55:33.318918 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:33.318929 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:33.320400 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:33.320405 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:33.322557 (+ 2152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:33.322572 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:33.324362 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:33.324368 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:33.324579 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:33.324616 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 02:55:33.325484 (+ 868us) client_negotiation.cc:240] Negotiation successful
0806 02:55:33.325649 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3930,"thread_start_us":4300,"threads_started":1}
I20260806 02:55:33.326897 14889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51456b87205647a7ae576e09acbe7017" candidate_uuid: "0cb38587aca8444fb40d67e32941ea03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c5c116a405d4e698eb9b15b3c50a17e" is_pre_election: true
I20260806 02:55:33.327431 14889 raft_consensus.cc:2454] T 51456b87205647a7ae576e09acbe7017 P 4c5c116a405d4e698eb9b15b3c50a17e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cb38587aca8444fb40d67e32941ea03 in term 0.
I20260806 02:55:33.327415 15324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51456b87205647a7ae576e09acbe7017" candidate_uuid: "0cb38587aca8444fb40d67e32941ea03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7acfb5f0395a47608eac1775aa86e11b" is_pre_election: true
I20260806 02:55:33.328136 15000 leader_election.cc:302] T 51456b87205647a7ae576e09acbe7017 P 0cb38587aca8444fb40d67e32941ea03 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cb38587aca8444fb40d67e32941ea03, 4c5c116a405d4e698eb9b15b3c50a17e; no voters:
I20260806 02:55:33.328190 15324 raft_consensus.cc:2454] T 51456b87205647a7ae576e09acbe7017 P 7acfb5f0395a47608eac1775aa86e11b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cb38587aca8444fb40d67e32941ea03 in term 0.
I20260806 02:55:33.328510 15428 raft_consensus.cc:2781] T 51456b87205647a7ae576e09acbe7017 P 0cb38587aca8444fb40d67e32941ea03 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:55:33.328640 15428 raft_consensus.cc:482] T 51456b87205647a7ae576e09acbe7017 P 0cb38587aca8444fb40d67e32941ea03 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:55:33.328732 15428 raft_consensus.cc:3037] T 51456b87205647a7ae576e09acbe7017 P 0cb38587aca8444fb40d67e32941ea03 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:55:33.331039 15441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:33.315579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.131:39655 (local address 127.25.211.129:56507)
0806 02:55:33.318751 (+ 3172us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:33.318771 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 02:55:33.318890 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:55:33.319935 (+ 1045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:55:33.319940 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:55:33.319966 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:55:33.320337 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:33.320350 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:33.321935 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:33.321940 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:33.324331 (+ 2391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:33.324348 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:33.330217 (+ 5869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:33.330223 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:33.330398 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:33.330433 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 02:55:33.330518 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 02:55:33.330683 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3033,"mutex_wait_us":3,"thread_start_us":9003,"threads_started":1}
I20260806 02:55:33.331888 15437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:33.316164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:56507 (local address 127.25.211.131:39655)
0806 02:55:33.319438 (+ 3274us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:33.319448 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:33.319470 (+ 22us) server_negotiation.cc:407] Connection header received
0806 02:55:33.319522 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:33.319526 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:33.319607 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:33.319696 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:55:33.320525 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:33.321774 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:33.326251 (+ 4477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:33.327715 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:33.327889 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:33.327958 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:33.331394 (+ 3436us) server_negotiation.cc:299] Negotiation successful
0806 02:55:33.331564 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3129,"thread_start_us":2952,"threads_started":1}
I20260806 02:55:33.332612 15410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:33.317798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:42763 (local address 127.25.211.130:32867)
0806 02:55:33.317979 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:33.317985 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:33.324971 (+ 6986us) server_negotiation.cc:407] Connection header received
0806 02:55:33.325226 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:33.325231 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:33.325311 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:33.325372 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:55:33.326204 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:33.327210 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:33.330089 (+ 2879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:33.331381 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:33.331523 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:33.331575 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:33.332224 (+ 649us) server_negotiation.cc:299] Negotiation successful
0806 02:55:33.332353 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260806 02:55:33.332567 15324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51456b87205647a7ae576e09acbe7017" candidate_uuid: "4c5c116a405d4e698eb9b15b3c50a17e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7acfb5f0395a47608eac1775aa86e11b" is_pre_election: true
I20260806 02:55:33.332850 15434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:33.316687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.130:32867 (local address 127.25.211.129:42763)
0806 02:55:33.324879 (+ 8192us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:33.324899 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 02:55:33.325081 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:55:33.325589 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:55:33.325593 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:55:33.325618 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:55:33.325952 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:33.325965 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:33.328125 (+ 2160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:33.328141 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:33.329933 (+ 1792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:33.329948 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:33.331589 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:33.331694 (+ 105us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:33.331867 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:33.331927 (+ 60us) client_negotiation.cc:769] Sending connection context
0806 02:55:33.332178 (+ 251us) client_negotiation.cc:240] Negotiation successful
0806 02:55:33.332314 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8058,"thread_start_us":1738,"threads_started":1}
I20260806 02:55:33.332851 15324 raft_consensus.cc:2454] T 51456b87205647a7ae576e09acbe7017 P 7acfb5f0395a47608eac1775aa86e11b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c5c116a405d4e698eb9b15b3c50a17e in term 0.
I20260806 02:55:33.332835 15428 raft_consensus.cc:504] T 51456b87205647a7ae576e09acbe7017 P 0cb38587aca8444fb40d67e32941ea03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c5c116a405d4e698eb9b15b3c50a17e" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 43767 } } peers { permanent_uuid: "7acfb5f0395a47608eac1775aa86e11b" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 39655 } } peers { permanent_uuid: "0cb38587aca8444fb40d67e32941ea03" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 32867 } }
I20260806 02:55:33.333428 14798 leader_election.cc:302] T 51456b87205647a7ae576e09acbe7017 P 4c5c116a405d4e698eb9b15b3c50a17e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c5c116a405d4e698eb9b15b3c50a17e, 7acfb5f0395a47608eac1775aa86e11b; no voters:
I20260806 02:55:33.333949 15422 raft_consensus.cc:2781] T 51456b87205647a7ae576e09acbe7017 P 4c5c116a405d4e698eb9b15b3c50a17e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:55:33.334096 15324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51456b87205647a7ae576e09acbe7017" candidate_uuid: "0cb38587aca8444fb40d67e32941ea03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7acfb5f0395a47608eac1775aa86e11b"
I20260806 02:55:33.334089 14889 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51456b87205647a7ae576e09acbe7017" candidate_uuid: "0cb38587aca8444fb40d67e32941ea03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c5c116a405d4e698eb9b15b3c50a17e"
I20260806 02:55:33.333940 15105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51456b87205647a7ae576e09acbe7017" candidate_uuid: "4c5c116a405d4e698eb9b15b3c50a17e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb38587aca8444fb40d67e32941ea03" is_pre_election: true
I20260806 02:55:33.334342 15324 raft_consensus.cc:3037] T 51456b87205647a7ae576e09acbe7017 P 7acfb5f0395a47608eac1775aa86e11b [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:55:33.334111 15422 raft_consensus.cc:482] T 51456b87205647a7ae576e09acbe7017 P 4c5c116a405d4e698eb9b15b3c50a17e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:55:33.334538 15105 raft_consensus.cc:2379] T 51456b87205647a7ae576e09acbe7017 P 0cb38587aca8444fb40d67e32941ea03 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4c5c116a405d4e698eb9b15b3c50a17e in current term 1: Already voted for candidate 0cb38587aca8444fb40d67e32941ea03 in this term.
I20260806 02:55:33.334522 15422 raft_consensus.cc:3037] T 51456b87205647a7ae576e09acbe7017 P 4c5c116a405d4e698eb9b15b3c50a17e [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:55:33.336818 15324 raft_consensus.cc:2454] T 51456b87205647a7ae576e09acbe7017 P 7acfb5f0395a47608eac1775aa86e11b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cb38587aca8444fb40d67e32941ea03 in term 1.
I20260806 02:55:33.337069 15428 leader_election.cc:288] T 51456b87205647a7ae576e09acbe7017 P 0cb38587aca8444fb40d67e32941ea03 [CANDIDATE]: Term 1 election: Requested vote from peers 4c5c116a405d4e698eb9b15b3c50a17e (127.25.211.129:43767), 7acfb5f0395a47608eac1775aa86e11b (127.25.211.131:39655)
I20260806 02:55:33.337064 15422 raft_consensus.cc:504] T 51456b87205647a7ae576e09acbe7017 P 4c5c116a405d4e698eb9b15b3c50a17e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c5c116a405d4e698eb9b15b3c50a17e" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 43767 } } peers { permanent_uuid: "7acfb5f0395a47608eac1775aa86e11b" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 39655 } } peers { permanent_uuid: "0cb38587aca8444fb40d67e32941ea03" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 32867 } }
I20260806 02:55:33.337240 15002 leader_election.cc:302] T 51456b87205647a7ae576e09acbe7017 P 0cb38587aca8444fb40d67e32941ea03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cb38587aca8444fb40d67e32941ea03, 7acfb5f0395a47608eac1775aa86e11b; no voters:
I20260806 02:55:33.337538 14889 raft_consensus.cc:2379] T 51456b87205647a7ae576e09acbe7017 P 4c5c116a405d4e698eb9b15b3c50a17e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0cb38587aca8444fb40d67e32941ea03 in current term 1: Already voted for candidate 4c5c116a405d4e698eb9b15b3c50a17e in this term.
I20260806 02:55:33.337538 15428 raft_consensus.cc:2781] T 51456b87205647a7ae576e09acbe7017 P 0cb38587aca8444fb40d67e32941ea03 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:55:33.337858 15422 leader_election.cc:288] T 51456b87205647a7ae576e09acbe7017 P 4c5c116a405d4e698eb9b15b3c50a17e [CANDIDATE]: Term 1 election: Requested vote from peers 7acfb5f0395a47608eac1775aa86e11b (127.25.211.131:39655), 0cb38587aca8444fb40d67e32941ea03 (127.25.211.130:32867)
I20260806 02:55:33.338250 15324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51456b87205647a7ae576e09acbe7017" candidate_uuid: "4c5c116a405d4e698eb9b15b3c50a17e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7acfb5f0395a47608eac1775aa86e11b"
I20260806 02:55:33.338536 15105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51456b87205647a7ae576e09acbe7017" candidate_uuid: "4c5c116a405d4e698eb9b15b3c50a17e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb38587aca8444fb40d67e32941ea03"
I20260806 02:55:33.338673 15324 raft_consensus.cc:2379] T 51456b87205647a7ae576e09acbe7017 P 7acfb5f0395a47608eac1775aa86e11b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4c5c116a405d4e698eb9b15b3c50a17e in current term 1: Already voted for candidate 0cb38587aca8444fb40d67e32941ea03 in this term.
I20260806 02:55:33.338884 15428 raft_consensus.cc:686] T 51456b87205647a7ae576e09acbe7017 P 0cb38587aca8444fb40d67e32941ea03 [term 1 LEADER]: Becoming Leader. State: Replica: 0cb38587aca8444fb40d67e32941ea03, State: Running, Role: LEADER
I20260806 02:55:33.339449 15428 consensus_queue.cc:237] T 51456b87205647a7ae576e09acbe7017 P 0cb38587aca8444fb40d67e32941ea03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c5c116a405d4e698eb9b15b3c50a17e" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 43767 } } peers { permanent_uuid: "7acfb5f0395a47608eac1775aa86e11b" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 39655 } } peers { permanent_uuid: "0cb38587aca8444fb40d67e32941ea03" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 32867 } }
I20260806 02:55:33.343245 14797 leader_election.cc:302] T 51456b87205647a7ae576e09acbe7017 P 4c5c116a405d4e698eb9b15b3c50a17e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c5c116a405d4e698eb9b15b3c50a17e; no voters: 0cb38587aca8444fb40d67e32941ea03, 7acfb5f0395a47608eac1775aa86e11b
I20260806 02:55:33.343722 15422 raft_consensus.cc:2726] T 51456b87205647a7ae576e09acbe7017 P 4c5c116a405d4e698eb9b15b3c50a17e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260806 02:55:33.344252 14677 catalog_manager.cc:5495] T 51456b87205647a7ae576e09acbe7017 P 0cb38587aca8444fb40d67e32941ea03 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0cb38587aca8444fb40d67e32941ea03 (127.25.211.130). New cstate: current_term: 1 leader_uuid: "0cb38587aca8444fb40d67e32941ea03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c5c116a405d4e698eb9b15b3c50a17e" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 43767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7acfb5f0395a47608eac1775aa86e11b" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 39655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cb38587aca8444fb40d67e32941ea03" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 32867 } health_report { overall_health: HEALTHY } } }
I20260806 02:55:33.362267 15410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:33.356937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35906 (local address 127.25.211.130:32867)
0806 02:55:33.357182 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:33.357188 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:33.357206 (+ 18us) server_negotiation.cc:407] Connection header received
0806 02:55:33.357331 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:33.357335 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:33.357413 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:33.357538 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:55:33.357887 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:33.358707 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:33.360170 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:33.360575 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:33.360675 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:33.361137 (+ 462us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:55:33.361644 (+ 507us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:55:33.361773 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:33.361847 (+ 74us) server_negotiation.cc:299] Negotiation successful
0806 02:55:33.361920 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
W20260806 02:55:33.366533 15181 tablet.cc:2369] T 51456b87205647a7ae576e09acbe7017 P 0cb38587aca8444fb40d67e32941ea03: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:55:33.370662 15324 raft_consensus.cc:1261] T 51456b87205647a7ae576e09acbe7017 P 7acfb5f0395a47608eac1775aa86e11b [term 1 FOLLOWER]: Refusing update from remote peer 0cb38587aca8444fb40d67e32941ea03: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:55:33.370704 14889 raft_consensus.cc:1261] T 51456b87205647a7ae576e09acbe7017 P 4c5c116a405d4e698eb9b15b3c50a17e [term 1 FOLLOWER]: Refusing update from remote peer 0cb38587aca8444fb40d67e32941ea03: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:55:33.371562 15442 consensus_queue.cc:1035] T 51456b87205647a7ae576e09acbe7017 P 0cb38587aca8444fb40d67e32941ea03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7acfb5f0395a47608eac1775aa86e11b" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 39655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:55:33.371917 15428 consensus_queue.cc:1035] T 51456b87205647a7ae576e09acbe7017 P 0cb38587aca8444fb40d67e32941ea03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c5c116a405d4e698eb9b15b3c50a17e" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 43767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:55:33.382418 15445 mvcc.cc:205] Tried to move back new op lower bound from 7315394287076372480 to 7315394286962237440. Current Snapshot: MvccSnapshot[applied={T|T < 7315394287076372480}]
I20260806 02:55:33.383280 15446 mvcc.cc:205] Tried to move back new op lower bound from 7315394287076372480 to 7315394286962237440. Current Snapshot: MvccSnapshot[applied={T|T < 7315394287076372480}]
I20260806 02:55:33.404358 14677 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43076:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:55:33.404666 14677 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43076:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:55:33.409116 14677 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4: Sending DeleteTablet for 3 replicas of tablet 51456b87205647a7ae576e09acbe7017
I20260806 02:55:33.410409 15290 tablet_service.cc:1514] Processing DeleteTablet for tablet 51456b87205647a7ae576e09acbe7017 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:55:33 UTC) from {username='kudu', principal='kudu/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:53768
I20260806 02:55:33.410409 14862 tablet_service.cc:1514] Processing DeleteTablet for tablet 51456b87205647a7ae576e09acbe7017 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:55:33 UTC) from {username='kudu', principal='kudu/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:60098
I20260806 02:55:33.411010 15077 tablet_service.cc:1514] Processing DeleteTablet for tablet 51456b87205647a7ae576e09acbe7017 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:55:33 UTC) from {username='kudu', principal='kudu/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:35904
I20260806 02:55:33.411870 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 14771
I20260806 02:55:33.422737 15455 tablet_replica.cc:330] T 51456b87205647a7ae576e09acbe7017 P 4c5c116a405d4e698eb9b15b3c50a17e: stopping tablet replica
I20260806 02:55:33.422809 15456 tablet_replica.cc:330] T 51456b87205647a7ae576e09acbe7017 P 0cb38587aca8444fb40d67e32941ea03: stopping tablet replica
I20260806 02:55:33.423316 15456 raft_consensus.cc:2229] T 51456b87205647a7ae576e09acbe7017 P 0cb38587aca8444fb40d67e32941ea03 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:55:33.423334 15455 raft_consensus.cc:2229] T 51456b87205647a7ae576e09acbe7017 P 4c5c116a405d4e698eb9b15b3c50a17e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:55:33.424015 15455 raft_consensus.cc:2258] T 51456b87205647a7ae576e09acbe7017 P 4c5c116a405d4e698eb9b15b3c50a17e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:55:33.424039 15456 raft_consensus.cc:2258] T 51456b87205647a7ae576e09acbe7017 P 0cb38587aca8444fb40d67e32941ea03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:55:33.424850 15454 tablet_replica.cc:330] T 51456b87205647a7ae576e09acbe7017 P 7acfb5f0395a47608eac1775aa86e11b: stopping tablet replica
I20260806 02:55:33.425215 15454 raft_consensus.cc:2229] T 51456b87205647a7ae576e09acbe7017 P 7acfb5f0395a47608eac1775aa86e11b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:55:33.425719 15454 raft_consensus.cc:2258] T 51456b87205647a7ae576e09acbe7017 P 7acfb5f0395a47608eac1775aa86e11b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:55:33.427671 15455 ts_tablet_manager.cc:1905] T 51456b87205647a7ae576e09acbe7017 P 4c5c116a405d4e698eb9b15b3c50a17e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:55:33.427690 15456 ts_tablet_manager.cc:1905] T 51456b87205647a7ae576e09acbe7017 P 0cb38587aca8444fb40d67e32941ea03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:55:33.433228 15454 ts_tablet_manager.cc:1905] T 51456b87205647a7ae576e09acbe7017 P 7acfb5f0395a47608eac1775aa86e11b: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 06 02:55:33 dist-test-slave-zj8q krb5kdc[14605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.129@KRBTEST.COM
I20260806 02:55:33.435508 15456 ts_tablet_manager.cc:1918] T 51456b87205647a7ae576e09acbe7017 P 0cb38587aca8444fb40d67e32941ea03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:55:33.435844 15456 log.cc:1196] T 51456b87205647a7ae576e09acbe7017 P 0cb38587aca8444fb40d67e32941ea03: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-1/wal/wals/51456b87205647a7ae576e09acbe7017
I20260806 02:55:33.436510 15456 ts_tablet_manager.cc:1939] T 51456b87205647a7ae576e09acbe7017 P 0cb38587aca8444fb40d67e32941ea03: Deleting consensus metadata
I20260806 02:55:33.438613 15455 ts_tablet_manager.cc:1918] T 51456b87205647a7ae576e09acbe7017 P 4c5c116a405d4e698eb9b15b3c50a17e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:55:33.438608 14652 catalog_manager.cc:4841] TS 0cb38587aca8444fb40d67e32941ea03 (127.25.211.130:32867): tablet 51456b87205647a7ae576e09acbe7017 (table test-table [id=ff4ff664d41b4f3b9d2cb0bbbee2f3e7]) successfully deleted
I20260806 02:55:33.438819 15455 log.cc:1196] T 51456b87205647a7ae576e09acbe7017 P 4c5c116a405d4e698eb9b15b3c50a17e: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-0/wal/wals/51456b87205647a7ae576e09acbe7017
I20260806 02:55:33.439330 15455 ts_tablet_manager.cc:1939] T 51456b87205647a7ae576e09acbe7017 P 4c5c116a405d4e698eb9b15b3c50a17e: Deleting consensus metadata
I20260806 02:55:33.440872 14649 catalog_manager.cc:4841] TS 4c5c116a405d4e698eb9b15b3c50a17e (127.25.211.129:43767): tablet 51456b87205647a7ae576e09acbe7017 (table test-table [id=ff4ff664d41b4f3b9d2cb0bbbee2f3e7]) successfully deleted
I20260806 02:55:33.443310 15406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:33.413226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60108 (local address 127.25.211.129:43767)
0806 02:55:33.413415 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:33.413421 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:33.420595 (+ 7174us) server_negotiation.cc:407] Connection header received
0806 02:55:33.420831 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:33.420838 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:33.420933 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:33.421065 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:33.424150 (+ 3085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:33.425342 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:33.427991 (+ 2649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:33.428679 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:33.428827 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:33.434910 (+ 6083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:33.434930 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:33.434950 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:33.435010 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:33.439705 (+ 4695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:33.440786 (+ 1081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:33.440797 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:33.440801 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:33.440962 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:33.441492 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:33.441497 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:33.441501 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:33.442025 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:33.442389 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:33.442711 (+ 322us) server_negotiation.cc:299] Negotiation successful
0806 02:55:33.442952 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 02:55:33.444060 14958 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:55:33.447347 15454 ts_tablet_manager.cc:1918] T 51456b87205647a7ae576e09acbe7017 P 7acfb5f0395a47608eac1775aa86e11b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:55:33.447535 15454 log.cc:1196] T 51456b87205647a7ae576e09acbe7017 P 7acfb5f0395a47608eac1775aa86e11b: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-2/wal/wals/51456b87205647a7ae576e09acbe7017
I20260806 02:55:33.448092 15454 ts_tablet_manager.cc:1939] T 51456b87205647a7ae576e09acbe7017 P 7acfb5f0395a47608eac1775aa86e11b: Deleting consensus metadata
I20260806 02:55:33.450083 14655 catalog_manager.cc:4841] TS 7acfb5f0395a47608eac1775aa86e11b (127.25.211.131:39655): tablet 51456b87205647a7ae576e09acbe7017 (table test-table [id=ff4ff664d41b4f3b9d2cb0bbbee2f3e7]) successfully deleted
I20260806 02:55:33.535869 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 14771
I20260806 02:55:33.559383 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 14971
Aug 06 02:55:33 dist-test-slave-zj8q krb5kdc[14605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.130@KRBTEST.COM
I20260806 02:55:33.577047 15410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:33.560135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35920 (local address 127.25.211.130:32867)
0806 02:55:33.560356 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:33.560362 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:33.560383 (+ 21us) server_negotiation.cc:407] Connection header received
0806 02:55:33.560596 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:33.560600 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:33.560674 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:33.560770 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:33.562070 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:33.562879 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:33.564207 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:33.564651 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:33.564768 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:33.569351 (+ 4583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:33.569369 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:33.569392 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:33.569450 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:33.573607 (+ 4157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:33.574551 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:33.574562 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:33.574566 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:33.574754 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:33.575228 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:33.575233 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:33.575236 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:33.575780 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:33.576096 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:33.576430 (+ 334us) server_negotiation.cc:299] Negotiation successful
0806 02:55:33.576672 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260806 02:55:33.577853 15175 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:55:33.692564 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 14971
I20260806 02:55:33.716503 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 15188
Aug 06 02:55:33 dist-test-slave-zj8q krb5kdc[14605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984930, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.131@KRBTEST.COM
I20260806 02:55:33.732690 15437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:33.717233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53774 (local address 127.25.211.131:39655)
0806 02:55:33.717393 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:33.717398 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:33.717440 (+ 42us) server_negotiation.cc:407] Connection header received
0806 02:55:33.717599 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:33.717603 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:33.717668 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:33.717761 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:33.719239 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:33.720009 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:33.721623 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:33.722013 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:33.722127 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:33.726618 (+ 4491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:33.726635 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:33.726653 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:33.726698 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:33.729731 (+ 3033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:33.730495 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:33.730505 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:33.730509 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:33.730650 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:33.731091 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:33.731096 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:33.731098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:33.731584 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:33.731833 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:33.732135 (+ 302us) server_negotiation.cc:299] Negotiation successful
0806 02:55:33.732361 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 02:55:33.733363 15378 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:55:33.824270 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 15188
I20260806 02:55:33.846498 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 14629
I20260806 02:55:33.847625 14721 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:55:33.960582 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 14629
I20260806 02:55:33.987015 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:45343
--webserver_interface=127.25.211.190
--webserver_port=34747
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:36275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.211.190:45343
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:55:34.334297 15470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:55:34.334666 15470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:55:34.334802 15470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:55:34.342478 15470 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:55:34.342628 15470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:55:34.342707 15470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:55:34.342784 15470 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:55:34.342845 15470 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:55:34.353410 15470 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:36275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.25.211.190:45343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:45343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.211.190
--webserver_port=34747
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:55:34.354893 15470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:55:34.356305 15470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:55:34.383119 15481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:34.383152 15478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:34.383188 15483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:55:34.384011 15470 server_base.cc:1034] running on GCE node
I20260806 02:55:34.384989 15470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:55:34.386809 15470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:55:34.388041 15470 hybrid_clock.cc:648] HybridClock initialized: now 1785984934387980 us; error 56 us; skew 500 ppm
I20260806 02:55:34.388470 15470 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:55:34 dist-test-slave-zj8q krb5kdc[14605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984934, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:55:34.397078 15470 init.cc:377] Logged in from keytab as kudu/127.25.211.190@KRBTEST.COM (short username kudu)
I20260806 02:55:34.399075 15470 webserver.cc:460] Webserver started at http://127.25.211.190:34747/ using document root <none> and password file <none>
I20260806 02:55:34.399883 15470 fs_manager.cc:359] Metadata directory not provided
I20260806 02:55:34.400030 15470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:55:34.406636 15470 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 02:55:34.409528 15493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:34.411074 15470 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:55:34.411242 15470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/master-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/master-0/wal
uuid: "7b628097688e47b7a0187a2f2f3a54e4"
format_stamp: "Formatted at 2026-08-06 02:55:31 on dist-test-slave-zj8q"
server_key: "4ef30fd60b93baa3a772901be83b1b44"
server_key_iv: "f0d33b683b62622baae7293d27b57725"
server_key_version: "encryptionkey@0"
I20260806 02:55:34.412329 15470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:55:34.428332 15470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:55:34.432518 15470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:55:34.432798 15470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:55:34.450654 15470 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.190:45343
I20260806 02:55:34.450678 15570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.190:45343 every 8 connection(s)
I20260806 02:55:34.452232 15470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
I20260806 02:55:34.459131 15572 sys_catalog.cc:263] Verifying existing consensus state
I20260806 02:55:34.462056 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 15470
I20260806 02:55:34.462551 15572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4: Bootstrap starting.
I20260806 02:55:34.462936 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.129:43767
--local_ip_for_outbound_sockets=127.25.211.129
--tserver_master_addrs=127.25.211.190:45343
--webserver_port=41871
--webserver_interface=127.25.211.129
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab
--principal=kudu/127.25.211.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:36275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:55:34.482708 15572 log.cc:824] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4: Log is configured to *not* fsync() on all Append() calls
I20260806 02:55:34.495141 15572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:55:34.495771 15572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4: Bootstrap complete.
I20260806 02:55:34.504992 15572 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b628097688e47b7a0187a2f2f3a54e4" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45343 } }
I20260806 02:55:34.505904 15572 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b628097688e47b7a0187a2f2f3a54e4, State: Initialized, Role: FOLLOWER
I20260806 02:55:34.506613 15572 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b628097688e47b7a0187a2f2f3a54e4" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45343 } }
I20260806 02:55:34.506844 15572 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:55:34.506971 15572 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:55:34.507123 15572 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4 [term 1 FOLLOWER]: Advancing to term 2
I20260806 02:55:34.511026 15572 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b628097688e47b7a0187a2f2f3a54e4" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45343 } }
I20260806 02:55:34.511543 15572 leader_election.cc:302] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b628097688e47b7a0187a2f2f3a54e4; no voters:
I20260806 02:55:34.512135 15572 leader_election.cc:288] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 02:55:34.512288 15577 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 02:55:34.513434 15577 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4 [term 2 LEADER]: Becoming Leader. State: Replica: 7b628097688e47b7a0187a2f2f3a54e4, State: Running, Role: LEADER
I20260806 02:55:34.513993 15577 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b628097688e47b7a0187a2f2f3a54e4" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45343 } }
I20260806 02:55:34.514463 15572 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:55:34.516530 15578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7b628097688e47b7a0187a2f2f3a54e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b628097688e47b7a0187a2f2f3a54e4" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45343 } } }
I20260806 02:55:34.517022 15578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4 [sys.catalog]: This master's current role is: LEADER
I20260806 02:55:34.517436 15582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b628097688e47b7a0187a2f2f3a54e4. Latest consensus state: current_term: 2 leader_uuid: "7b628097688e47b7a0187a2f2f3a54e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b628097688e47b7a0187a2f2f3a54e4" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45343 } } }
I20260806 02:55:34.517724 15582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4 [sys.catalog]: This master's current role is: LEADER
I20260806 02:55:34.522257 15588 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:55:34.531613 15588 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:55:34.533227 15588 catalog_manager.cc:1259] Loaded cluster ID: a5f8fb30977e47c9b2037092c1b5bb2d
I20260806 02:55:34.533349 15588 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:55:34.536839 15588 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260806 02:55:34.555244 15588 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260806 02:55:34.573680 15588 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:55:34.575232 15588 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4: Loaded TSK: 0
I20260806 02:55:34.576545 15588 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 02:55:34.846253 15575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:55:34.846570 15575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:55:34.846693 15575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:55:34.854172 15575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:55:34.854313 15575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:55:34.854475 15575 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.129
I20260806 02:55:34.865097 15575 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:36275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab
--principal=kudu/127.25.211.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.129:43767
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.211.129
--webserver_port=41871
--tserver_master_addrs=127.25.211.190:45343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.129
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:55:34.866573 15575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:55:34.867995 15575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:55:34.881448 15616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:34.881479 15620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:55:34.882819 15575 server_base.cc:1034] running on GCE node
W20260806 02:55:34.882850 15624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:55:34.883751 15575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:55:34.885006 15575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:55:34.886268 15575 hybrid_clock.cc:648] HybridClock initialized: now 1785984934886196 us; error 61 us; skew 500 ppm
I20260806 02:55:34.886672 15575 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:55:34 dist-test-slave-zj8q krb5kdc[14605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984934, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:55:34.890656 15575 init.cc:377] Logged in from keytab as kudu/127.25.211.129@KRBTEST.COM (short username kudu)
I20260806 02:55:34.892288 15575 webserver.cc:460] Webserver started at http://127.25.211.129:41871/ using document root <none> and password file <none>
I20260806 02:55:34.893021 15575 fs_manager.cc:359] Metadata directory not provided
I20260806 02:55:34.893162 15575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:55:34.899156 15575 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 02:55:34.902058 15640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:34.903684 15575 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 02:55:34.903879 15575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-0/wal
uuid: "4c5c116a405d4e698eb9b15b3c50a17e"
format_stamp: "Formatted at 2026-08-06 02:55:31 on dist-test-slave-zj8q"
server_key: "3aa6a7f11d98841dc43f0e707e8bc873"
server_key_iv: "fce77d93f36ef050aebc4890ec034c96"
server_key_version: "encryptionkey@0"
I20260806 02:55:34.904882 15575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:55:34.932178 15575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:55:34.936420 15575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:55:34.936731 15575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:55:34.937880 15575 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:55:34.939356 15575 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:55:34.939462 15575 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:34.939584 15575 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:55:34.939666 15575 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:34.971899 15575 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.129:43767
I20260806 02:55:34.972000 15837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.129:43767 every 8 connection(s)
I20260806 02:55:34.973421 15575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
I20260806 02:55:34.983448 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 15575
I20260806 02:55:34.984098 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.130:32867
--local_ip_for_outbound_sockets=127.25.211.130
--tserver_master_addrs=127.25.211.190:45343
--webserver_port=37671
--webserver_interface=127.25.211.130
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab
--principal=kudu/127.25.211.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:36275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:55:34 dist-test-slave-zj8q krb5kdc[14605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984934, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.129@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:55:35.003795 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:34.976455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:46807 (local address 127.25.211.190:45343)
0806 02:55:34.977108 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:34.977126 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:34.977632 (+ 506us) server_negotiation.cc:407] Connection header received
0806 02:55:34.979917 (+ 2285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:34.979932 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:34.980307 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:34.980622 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:34.982135 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:34.983497 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:34.985308 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:34.985991 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:34.986222 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:34.996243 (+ 10021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:34.996253 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:34.996271 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:34.996316 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:34.999197 (+ 2881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:35.000192 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:35.000200 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:35.000206 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:35.000388 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:35.000915 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:35.000918 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:35.000920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:35.001426 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:35.001583 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:35.002468 (+ 885us) server_negotiation.cc:299] Negotiation successful
0806 02:55:35.002811 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":242,"threads_started":1}
I20260806 02:55:35.004081 15844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:34.976649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45343 (local address 127.25.211.129:46807)
0806 02:55:34.977451 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:34.977490 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 02:55:34.979349 (+ 1859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:55:34.980773 (+ 1424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:55:34.980781 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:55:34.981329 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:55:34.981918 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:34.981937 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:34.983692 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:34.983696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:34.985134 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:34.985149 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:34.986284 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:34.986293 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:34.986560 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:34.993162 (+ 6602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:55:34.993200 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:55:34.995866 (+ 2666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:55:34.999379 (+ 3513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:34.999390 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:34.999406 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:35.000040 (+ 634us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:35.000518 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:35.000523 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:35.000526 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:35.000807 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:35.001576 (+ 769us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:55:35.001584 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:55:35.001978 (+ 394us) client_negotiation.cc:769] Sending connection context
0806 02:55:35.002313 (+ 335us) client_negotiation.cc:240] Negotiation successful
0806 02:55:35.002917 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":313,"threads_started":1}
I20260806 02:55:35.007282 15838 heartbeater.cc:347] Connected to a master server at 127.25.211.190:45343
I20260806 02:55:35.007653 15838 heartbeater.cc:464] Registering TS with master...
I20260806 02:55:35.008646 15838 heartbeater.cc:511] Master 127.25.211.190:45343 requested a full tablet report, sending...
I20260806 02:55:35.011417 15515 ts_manager.cc:194] Registered new tserver with Master: 4c5c116a405d4e698eb9b15b3c50a17e (127.25.211.129:43767)
I20260806 02:55:35.013782 15515 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.211.129@KRBTEST.COM'} at 127.25.211.129:46807
W20260806 02:55:35.343593 15847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:55:35.343914 15847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:55:35.344033 15847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:55:35.351506 15847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:55:35.351651 15847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:55:35.351799 15847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.130
I20260806 02:55:35.361975 15847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:36275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab
--principal=kudu/127.25.211.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.130:32867
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.211.130
--webserver_port=37671
--tserver_master_addrs=127.25.211.190:45343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.130
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:55:35.363426 15847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:55:35.364876 15847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:55:35.378733 15853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:35.378816 15858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:35.378764 15854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:55:35.379807 15847 server_base.cc:1034] running on GCE node
I20260806 02:55:35.380506 15847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:55:35.381551 15847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:55:35.382799 15847 hybrid_clock.cc:648] HybridClock initialized: now 1785984935382734 us; error 53 us; skew 500 ppm
I20260806 02:55:35.383176 15847 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:55:35 dist-test-slave-zj8q krb5kdc[14605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984935, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:55:35.386569 15847 init.cc:377] Logged in from keytab as kudu/127.25.211.130@KRBTEST.COM (short username kudu)
I20260806 02:55:35.388151 15847 webserver.cc:460] Webserver started at http://127.25.211.130:37671/ using document root <none> and password file <none>
I20260806 02:55:35.388902 15847 fs_manager.cc:359] Metadata directory not provided
I20260806 02:55:35.389053 15847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:55:35.395133 15847 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 02:55:35.398734 15869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:35.400686 15847 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260806 02:55:35.400868 15847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-1/wal
uuid: "0cb38587aca8444fb40d67e32941ea03"
format_stamp: "Formatted at 2026-08-06 02:55:32 on dist-test-slave-zj8q"
server_key: "98e38b2e4a4e0b25bca5309f3b32bde0"
server_key_iv: "1384851389e32776e8555760b7c71c14"
server_key_version: "encryptionkey@0"
I20260806 02:55:35.401942 15847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:55:35.428102 15847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:55:35.432200 15847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:55:35.432456 15847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:55:35.433460 15847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:55:35.434875 15847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:55:35.434973 15847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:35.435083 15847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:55:35.435144 15847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:35.467525 15847 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.130:32867
I20260806 02:55:35.467607 16050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.130:32867 every 8 connection(s)
I20260806 02:55:35.468928 15847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
I20260806 02:55:35.476884 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 15847
I20260806 02:55:35.477501 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.131:39655
--local_ip_for_outbound_sockets=127.25.211.131
--tserver_master_addrs=127.25.211.190:45343
--webserver_port=42341
--webserver_interface=127.25.211.131
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab
--principal=kudu/127.25.211.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:36275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:55:35 dist-test-slave-zj8q krb5kdc[14605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984935, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.130@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:55:35.489225 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:35.471604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:34233 (local address 127.25.211.190:45343)
0806 02:55:35.471786 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:35.471791 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:35.472946 (+ 1155us) server_negotiation.cc:407] Connection header received
0806 02:55:35.474782 (+ 1836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:35.474787 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:35.474869 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:35.474966 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:35.476465 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:35.477209 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:35.478880 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:35.479324 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:35.479442 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:35.482970 (+ 3528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:35.482975 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:35.482978 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:35.483014 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:35.485645 (+ 2631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:35.486645 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:35.486650 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:35.486652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:35.486823 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:35.487298 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:35.487302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:35.487304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:35.487606 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:35.487775 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:35.488576 (+ 801us) server_negotiation.cc:299] Negotiation successful
0806 02:55:35.488866 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 02:55:35.490345 16054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:35.472029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45343 (local address 127.25.211.130:34233)
0806 02:55:35.472775 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:35.472820 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 02:55:35.474532 (+ 1712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:55:35.475149 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:55:35.475158 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:55:35.475694 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:55:35.476258 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:35.476276 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:35.477375 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:35.477379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:35.478725 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:35.478738 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:35.479549 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:35.479557 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:35.479785 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:35.480425 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:55:35.480448 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:55:35.482711 (+ 2263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:55:35.485874 (+ 3163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:35.485885 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:35.485901 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:35.486491 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:35.486947 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:35.486951 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:35.486953 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:35.487170 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:35.487758 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:55:35.487768 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:55:35.488158 (+ 390us) client_negotiation.cc:769] Sending connection context
0806 02:55:35.488429 (+ 271us) client_negotiation.cc:240] Negotiation successful
0806 02:55:35.489029 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":284,"threads_started":1}
I20260806 02:55:35.491932 16051 heartbeater.cc:347] Connected to a master server at 127.25.211.190:45343
I20260806 02:55:35.492298 16051 heartbeater.cc:464] Registering TS with master...
I20260806 02:55:35.493249 16051 heartbeater.cc:511] Master 127.25.211.190:45343 requested a full tablet report, sending...
I20260806 02:55:35.495065 15515 ts_manager.cc:194] Registered new tserver with Master: 0cb38587aca8444fb40d67e32941ea03 (127.25.211.130:32867)
I20260806 02:55:35.496186 15515 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.211.130@KRBTEST.COM'} at 127.25.211.130:34233
W20260806 02:55:35.827024 16055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:55:35.827335 16055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:55:35.827430 16055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:55:35.834944 16055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:55:35.835083 16055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:55:35.835204 16055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.131
I20260806 02:55:35.845666 16055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:36275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab
--principal=kudu/127.25.211.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.131:39655
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.211.131
--webserver_port=42341
--tserver_master_addrs=127.25.211.190:45343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.131
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:55:35.847114 16055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:55:35.848482 16055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:55:35.865487 16062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:35.866240 16063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:35.866297 16065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:55:35.867985 16055 server_base.cc:1034] running on GCE node
I20260806 02:55:35.868675 16055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:55:35.869757 16055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:55:35.870961 16055 hybrid_clock.cc:648] HybridClock initialized: now 1785984935870900 us; error 53 us; skew 500 ppm
I20260806 02:55:35.871279 16055 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:55:35 dist-test-slave-zj8q krb5kdc[14605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984935, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:55:35.874420 16055 init.cc:377] Logged in from keytab as kudu/127.25.211.131@KRBTEST.COM (short username kudu)
I20260806 02:55:35.875898 16055 webserver.cc:460] Webserver started at http://127.25.211.131:42341/ using document root <none> and password file <none>
I20260806 02:55:35.876597 16055 fs_manager.cc:359] Metadata directory not provided
I20260806 02:55:35.876736 16055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:55:35.882405 16055 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 02:55:35.885108 16082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:35.886507 16055 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:55:35.886682 16055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-2/wal
uuid: "7acfb5f0395a47608eac1775aa86e11b"
format_stamp: "Formatted at 2026-08-06 02:55:33 on dist-test-slave-zj8q"
server_key: "9aa9fb40458463c7dd8c14404dab453e"
server_key_iv: "cb04e582455cef0709d5070676529619"
server_key_version: "encryptionkey@0"
I20260806 02:55:35.887646 16055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:55:35.912523 16055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:55:35.916877 16055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:55:35.917145 16055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:55:35.918193 16055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:55:35.919607 16055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:55:35.919725 16055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:35.919832 16055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:55:35.919901 16055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:35.952105 16055 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.131:39655
I20260806 02:55:35.952178 16238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.131:39655 every 8 connection(s)
I20260806 02:55:35.953624 16055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
I20260806 02:55:35.961241 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 16055
Aug 06 02:55:35 dist-test-slave-zj8q krb5kdc[14605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984935, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.131@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:55:35.974699 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:35.956236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:36919 (local address 127.25.211.190:45343)
0806 02:55:35.956414 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:35.956418 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:35.957517 (+ 1099us) server_negotiation.cc:407] Connection header received
0806 02:55:35.959393 (+ 1876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:35.959398 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:35.959470 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:35.959558 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:35.961091 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:35.962095 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:35.964241 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:35.964676 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:35.964779 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:35.968586 (+ 3807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:35.968591 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:35.968596 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:35.968629 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:35.971104 (+ 2475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:35.972011 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:35.972017 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:35.972020 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:35.972205 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:35.972746 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:35.972752 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:35.972756 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:35.973040 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:35.973156 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:35.974042 (+ 886us) server_negotiation.cc:299] Negotiation successful
0806 02:55:35.974330 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 02:55:35.975823 16242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:35.956638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45343 (local address 127.25.211.131:36919)
0806 02:55:35.957347 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:35.957386 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 02:55:35.959113 (+ 1727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:55:35.959787 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:55:35.959796 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:55:35.960320 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:55:35.960907 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:35.960925 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:35.962293 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:35.962297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:35.964094 (+ 1797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:35.964109 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:35.964903 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:35.964912 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:35.965199 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:35.965909 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:55:35.965932 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:55:35.968353 (+ 2421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:55:35.971245 (+ 2892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:35.971253 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:35.971269 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:35.971873 (+ 604us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:35.972329 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:35.972335 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:35.972339 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:35.972577 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:35.973168 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:55:35.973175 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:55:35.973614 (+ 439us) client_negotiation.cc:769] Sending connection context
0806 02:55:35.973923 (+ 309us) client_negotiation.cc:240] Negotiation successful
0806 02:55:35.974502 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":264,"threads_started":1}
I20260806 02:55:35.977317 16240 heartbeater.cc:347] Connected to a master server at 127.25.211.190:45343
I20260806 02:55:35.977648 16240 heartbeater.cc:464] Registering TS with master...
I20260806 02:55:35.978475 16240 heartbeater.cc:511] Master 127.25.211.190:45343 requested a full tablet report, sending...
I20260806 02:55:35.979344 16245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:35.962258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43090 (local address 127.25.211.190:45343)
0806 02:55:35.962773 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:35.962778 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:35.963269 (+ 491us) server_negotiation.cc:407] Connection header received
0806 02:55:35.963473 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:35.963477 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:35.963560 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:35.963653 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:35.965528 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:35.966321 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:35.967897 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:35.968349 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:35.968442 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:35.971052 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:35.971059 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:35.971064 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:35.971097 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:35.974605 (+ 3508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:35.977319 (+ 2714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:35.977324 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:35.977326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:35.977474 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:35.977888 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:35.977892 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:35.977894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:35.978207 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:35.978445 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:35.978760 (+ 315us) server_negotiation.cc:299] Negotiation successful
0806 02:55:35.978982 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":256,"threads_started":1}
I20260806 02:55:35.980062 15515 ts_manager.cc:194] Registered new tserver with Master: 7acfb5f0395a47608eac1775aa86e11b (127.25.211.131:39655)
I20260806 02:55:35.980875 15515 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.211.131@KRBTEST.COM'} at 127.25.211.131:36919
I20260806 02:55:35.981968 26446 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:55:35.988770 16245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:35.982913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43104 (local address 127.25.211.190:45343)
0806 02:55:35.983053 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:35.983057 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:35.983575 (+ 518us) server_negotiation.cc:407] Connection header received
0806 02:55:35.983790 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:35.983794 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:35.983921 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:35.984041 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:55:35.984463 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:35.985142 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:35.986715 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:35.987127 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:35.987248 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:35.987713 (+ 465us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:55:35.988197 (+ 484us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:55:35.988334 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:35.988447 (+ 113us) server_negotiation.cc:299] Negotiation successful
0806 02:55:35.988526 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 02:55:35.989698 15514 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:43104:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:55:35.997455 15514 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:55:36.018257 15838 heartbeater.cc:503] Master 127.25.211.190:45343 was elected leader, sending a full tablet report...
I20260806 02:55:36.033360 16261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:36.010958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60110 (local address 127.25.211.129:43767)
0806 02:55:36.023696 (+ 12738us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:36.023706 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:36.023738 (+ 32us) server_negotiation.cc:407] Connection header received
0806 02:55:36.023841 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:36.023849 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:36.024084 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:36.025040 (+ 956us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:55:36.025293 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:36.027049 (+ 1756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:36.030345 (+ 3296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:36.031914 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:36.032609 (+ 695us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:36.032783 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:36.032857 (+ 74us) server_negotiation.cc:299] Negotiation successful
0806 02:55:36.033024 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12551,"thread_start_us":12331,"threads_started":1}
I20260806 02:55:36.033380 16258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:36.010236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:43767 (local address 127.0.0.1:60110)
0806 02:55:36.022735 (+ 12499us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:36.022760 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 02:55:36.023022 (+ 262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:55:36.024379 (+ 1357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:55:36.024385 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:55:36.024416 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:55:36.024838 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:36.024850 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:36.027242 (+ 2392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:36.027248 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:36.029792 (+ 2544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:36.029807 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:36.032162 (+ 2355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:36.032171 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:36.032438 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:36.032591 (+ 153us) client_negotiation.cc:769] Sending connection context
0806 02:55:36.032730 (+ 139us) client_negotiation.cc:240] Negotiation successful
0806 02:55:36.032950 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12323,"mutex_wait_us":49,"thread_start_us":931,"threads_started":1}
I20260806 02:55:36.033972 16259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:36.010030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35930 (local address 127.25.211.130:32867)
0806 02:55:36.022367 (+ 12337us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:36.022379 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:36.022421 (+ 42us) server_negotiation.cc:407] Connection header received
0806 02:55:36.022558 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:36.022568 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:36.022896 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:36.024520 (+ 1624us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:55:36.024771 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:36.026508 (+ 1737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:36.030608 (+ 4100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:36.032057 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:36.032219 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:36.032422 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:36.033514 (+ 1092us) server_negotiation.cc:299] Negotiation successful
0806 02:55:36.033682 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12056,"thread_start_us":11931,"threads_started":1}
I20260806 02:55:36.034190 16260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:36.009835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.130:32867 (local address 127.0.0.1:35930)
0806 02:55:36.011125 (+ 1290us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:36.011179 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 02:55:36.011370 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:55:36.023507 (+ 12137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:55:36.023517 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:55:36.023560 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:55:36.024046 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:36.024068 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:36.026721 (+ 2653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:36.026727 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:36.030421 (+ 3694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:36.030437 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:36.032339 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:36.032346 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:36.032537 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:36.032590 (+ 53us) client_negotiation.cc:769] Sending connection context
0806 02:55:36.033182 (+ 592us) client_negotiation.cc:240] Negotiation successful
0806 02:55:36.033324 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":979,"mutex_wait_us":1022,"thread_start_us":13246,"threads_started":1}
I20260806 02:55:36.037200 15718 tablet_service.cc:1467] Processing CreateTablet for tablet e32d0cb55d354d37a1706a65ab759a7b (DEFAULT_TABLE table=test-table [id=6d12b9c82ddf427691ff81eeeabfef1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:55:36.037583 15949 tablet_service.cc:1467] Processing CreateTablet for tablet e32d0cb55d354d37a1706a65ab759a7b (DEFAULT_TABLE table=test-table [id=6d12b9c82ddf427691ff81eeeabfef1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:55:36.038787 15718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e32d0cb55d354d37a1706a65ab759a7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:55:36.038985 15949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e32d0cb55d354d37a1706a65ab759a7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:55:36.040483 16262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:36.012088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.131:39655 (local address 127.0.0.1:53780)
0806 02:55:36.029783 (+ 17695us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:36.029807 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 02:55:36.029945 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:55:36.032214 (+ 2269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:55:36.032219 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:55:36.032251 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:55:36.032832 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:36.032846 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:36.035727 (+ 2881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:36.035732 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:36.037695 (+ 1963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:36.037715 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:36.039586 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:36.039593 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:36.039796 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:36.039830 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 02:55:36.039931 (+ 101us) client_negotiation.cc:240] Negotiation successful
0806 02:55:36.040073 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17520,"mutex_wait_us":44,"thread_start_us":17324,"threads_started":1}
I20260806 02:55:36.040520 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:36.012496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53780 (local address 127.25.211.131:39655)
0806 02:55:36.031326 (+ 18830us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:36.031336 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:36.031373 (+ 37us) server_negotiation.cc:407] Connection header received
0806 02:55:36.031477 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:36.031485 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:36.031772 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:36.033037 (+ 1265us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:55:36.033322 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:36.035561 (+ 2239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:36.037903 (+ 2342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:36.039352 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:36.039553 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:36.039743 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:36.039993 (+ 250us) server_negotiation.cc:299] Negotiation successful
0806 02:55:36.040166 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12205,"thread_start_us":12967,"threads_started":1}
I20260806 02:55:36.044981 16148 tablet_service.cc:1467] Processing CreateTablet for tablet e32d0cb55d354d37a1706a65ab759a7b (DEFAULT_TABLE table=test-table [id=6d12b9c82ddf427691ff81eeeabfef1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:55:36.046591 16148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e32d0cb55d354d37a1706a65ab759a7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:55:36.052899 16264 tablet_bootstrap.cc:492] T e32d0cb55d354d37a1706a65ab759a7b P 4c5c116a405d4e698eb9b15b3c50a17e: Bootstrap starting.
I20260806 02:55:36.054925 16266 tablet_bootstrap.cc:492] T e32d0cb55d354d37a1706a65ab759a7b P 0cb38587aca8444fb40d67e32941ea03: Bootstrap starting.
I20260806 02:55:36.056815 16264 tablet_bootstrap.cc:654] T e32d0cb55d354d37a1706a65ab759a7b P 4c5c116a405d4e698eb9b15b3c50a17e: Neither blocks nor log segments found. Creating new log.
I20260806 02:55:36.058498 16266 tablet_bootstrap.cc:654] T e32d0cb55d354d37a1706a65ab759a7b P 0cb38587aca8444fb40d67e32941ea03: Neither blocks nor log segments found. Creating new log.
I20260806 02:55:36.059060 16264 log.cc:824] T e32d0cb55d354d37a1706a65ab759a7b P 4c5c116a405d4e698eb9b15b3c50a17e: Log is configured to *not* fsync() on all Append() calls
I20260806 02:55:36.059755 16266 log.cc:824] T e32d0cb55d354d37a1706a65ab759a7b P 0cb38587aca8444fb40d67e32941ea03: Log is configured to *not* fsync() on all Append() calls
I20260806 02:55:36.061576 16276 tablet_bootstrap.cc:492] T e32d0cb55d354d37a1706a65ab759a7b P 7acfb5f0395a47608eac1775aa86e11b: Bootstrap starting.
I20260806 02:55:36.062876 16264 tablet_bootstrap.cc:492] T e32d0cb55d354d37a1706a65ab759a7b P 4c5c116a405d4e698eb9b15b3c50a17e: No bootstrap required, opened a new log
I20260806 02:55:36.063191 16264 ts_tablet_manager.cc:1397] T e32d0cb55d354d37a1706a65ab759a7b P 4c5c116a405d4e698eb9b15b3c50a17e: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260806 02:55:36.063235 16266 tablet_bootstrap.cc:492] T e32d0cb55d354d37a1706a65ab759a7b P 0cb38587aca8444fb40d67e32941ea03: No bootstrap required, opened a new log
I20260806 02:55:36.063499 16266 ts_tablet_manager.cc:1397] T e32d0cb55d354d37a1706a65ab759a7b P 0cb38587aca8444fb40d67e32941ea03: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260806 02:55:36.064968 16276 tablet_bootstrap.cc:654] T e32d0cb55d354d37a1706a65ab759a7b P 7acfb5f0395a47608eac1775aa86e11b: Neither blocks nor log segments found. Creating new log.
I20260806 02:55:36.066272 16276 log.cc:824] T e32d0cb55d354d37a1706a65ab759a7b P 7acfb5f0395a47608eac1775aa86e11b: Log is configured to *not* fsync() on all Append() calls
I20260806 02:55:36.068878 16276 tablet_bootstrap.cc:492] T e32d0cb55d354d37a1706a65ab759a7b P 7acfb5f0395a47608eac1775aa86e11b: No bootstrap required, opened a new log
I20260806 02:55:36.069139 16276 ts_tablet_manager.cc:1397] T e32d0cb55d354d37a1706a65ab759a7b P 7acfb5f0395a47608eac1775aa86e11b: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260806 02:55:36.074021 16266 raft_consensus.cc:348] T e32d0cb55d354d37a1706a65ab759a7b P 0cb38587aca8444fb40d67e32941ea03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb38587aca8444fb40d67e32941ea03" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 32867 } } peers { permanent_uuid: "4c5c116a405d4e698eb9b15b3c50a17e" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 43767 } } peers { permanent_uuid: "7acfb5f0395a47608eac1775aa86e11b" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 39655 } }
I20260806 02:55:36.074021 16264 raft_consensus.cc:348] T e32d0cb55d354d37a1706a65ab759a7b P 4c5c116a405d4e698eb9b15b3c50a17e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb38587aca8444fb40d67e32941ea03" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 32867 } } peers { permanent_uuid: "4c5c116a405d4e698eb9b15b3c50a17e" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 43767 } } peers { permanent_uuid: "7acfb5f0395a47608eac1775aa86e11b" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 39655 } }
I20260806 02:55:36.074543 16266 raft_consensus.cc:374] T e32d0cb55d354d37a1706a65ab759a7b P 0cb38587aca8444fb40d67e32941ea03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:55:36.074543 16264 raft_consensus.cc:374] T e32d0cb55d354d37a1706a65ab759a7b P 4c5c116a405d4e698eb9b15b3c50a17e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:55:36.074651 16264 raft_consensus.cc:729] T e32d0cb55d354d37a1706a65ab759a7b P 4c5c116a405d4e698eb9b15b3c50a17e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c5c116a405d4e698eb9b15b3c50a17e, State: Initialized, Role: FOLLOWER
I20260806 02:55:36.074648 16266 raft_consensus.cc:729] T e32d0cb55d354d37a1706a65ab759a7b P 0cb38587aca8444fb40d67e32941ea03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cb38587aca8444fb40d67e32941ea03, State: Initialized, Role: FOLLOWER
I20260806 02:55:36.075412 16264 consensus_queue.cc:260] T e32d0cb55d354d37a1706a65ab759a7b P 4c5c116a405d4e698eb9b15b3c50a17e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb38587aca8444fb40d67e32941ea03" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 32867 } } peers { permanent_uuid: "4c5c116a405d4e698eb9b15b3c50a17e" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 43767 } } peers { permanent_uuid: "7acfb5f0395a47608eac1775aa86e11b" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 39655 } }
I20260806 02:55:36.075412 16266 consensus_queue.cc:260] T e32d0cb55d354d37a1706a65ab759a7b P 0cb38587aca8444fb40d67e32941ea03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb38587aca8444fb40d67e32941ea03" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 32867 } } peers { permanent_uuid: "4c5c116a405d4e698eb9b15b3c50a17e" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 43767 } } peers { permanent_uuid: "7acfb5f0395a47608eac1775aa86e11b" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 39655 } }
I20260806 02:55:36.076777 16276 raft_consensus.cc:348] T e32d0cb55d354d37a1706a65ab759a7b P 7acfb5f0395a47608eac1775aa86e11b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb38587aca8444fb40d67e32941ea03" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 32867 } } peers { permanent_uuid: "4c5c116a405d4e698eb9b15b3c50a17e" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 43767 } } peers { permanent_uuid: "7acfb5f0395a47608eac1775aa86e11b" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 39655 } }
I20260806 02:55:36.077229 16276 raft_consensus.cc:374] T e32d0cb55d354d37a1706a65ab759a7b P 7acfb5f0395a47608eac1775aa86e11b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:55:36.077327 16276 raft_consensus.cc:729] T e32d0cb55d354d37a1706a65ab759a7b P 7acfb5f0395a47608eac1775aa86e11b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7acfb5f0395a47608eac1775aa86e11b, State: Initialized, Role: FOLLOWER
I20260806 02:55:36.077368 16266 ts_tablet_manager.cc:1428] T e32d0cb55d354d37a1706a65ab759a7b P 0cb38587aca8444fb40d67e32941ea03: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260806 02:55:36.077700 16264 ts_tablet_manager.cc:1428] T e32d0cb55d354d37a1706a65ab759a7b P 4c5c116a405d4e698eb9b15b3c50a17e: Time spent starting tablet: real 0.014s user 0.010s sys 0.005s
I20260806 02:55:36.077919 16051 heartbeater.cc:503] Master 127.25.211.190:45343 was elected leader, sending a full tablet report...
I20260806 02:55:36.078065 16276 consensus_queue.cc:260] T e32d0cb55d354d37a1706a65ab759a7b P 7acfb5f0395a47608eac1775aa86e11b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb38587aca8444fb40d67e32941ea03" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 32867 } } peers { permanent_uuid: "4c5c116a405d4e698eb9b15b3c50a17e" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 43767 } } peers { permanent_uuid: "7acfb5f0395a47608eac1775aa86e11b" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 39655 } }
I20260806 02:55:36.079298 16240 heartbeater.cc:503] Master 127.25.211.190:45343 was elected leader, sending a full tablet report...
I20260806 02:55:36.080005 16276 ts_tablet_manager.cc:1428] T e32d0cb55d354d37a1706a65ab759a7b P 7acfb5f0395a47608eac1775aa86e11b: Time spent starting tablet: real 0.011s user 0.007s sys 0.004s
I20260806 02:55:36.134768 16280 raft_consensus.cc:482] T e32d0cb55d354d37a1706a65ab759a7b P 0cb38587aca8444fb40d67e32941ea03 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:55:36.135165 16280 raft_consensus.cc:504] T e32d0cb55d354d37a1706a65ab759a7b P 0cb38587aca8444fb40d67e32941ea03 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb38587aca8444fb40d67e32941ea03" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 32867 } } peers { permanent_uuid: "4c5c116a405d4e698eb9b15b3c50a17e" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 43767 } } peers { permanent_uuid: "7acfb5f0395a47608eac1775aa86e11b" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 39655 } }
I20260806 02:55:36.136895 16280 leader_election.cc:288] T e32d0cb55d354d37a1706a65ab759a7b P 0cb38587aca8444fb40d67e32941ea03 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c5c116a405d4e698eb9b15b3c50a17e (127.25.211.129:43767), 7acfb5f0395a47608eac1775aa86e11b (127.25.211.131:39655)
I20260806 02:55:36.143684 16261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:36.137365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:50627 (local address 127.25.211.129:43767)
0806 02:55:36.137585 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:36.137591 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:36.138318 (+ 727us) server_negotiation.cc:407] Connection header received
0806 02:55:36.138521 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:36.138525 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:36.138585 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:36.138659 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:55:36.139234 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:36.139964 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:36.141931 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:36.142733 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:36.142886 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:36.142962 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:36.143298 (+ 336us) server_negotiation.cc:299] Negotiation successful
0806 02:55:36.143439 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 02:55:36.143730 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:36.137361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:56723 (local address 127.25.211.131:39655)
0806 02:55:36.137570 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:36.137577 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:36.138437 (+ 860us) server_negotiation.cc:407] Connection header received
0806 02:55:36.138617 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:36.138621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:36.138681 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:36.138754 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:55:36.139382 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:36.140093 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:36.141920 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:36.142774 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:36.142925 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:36.142998 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:36.143323 (+ 325us) server_negotiation.cc:299] Negotiation successful
0806 02:55:36.143451 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 02:55:36.143823 16283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:36.137387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:43767 (local address 127.25.211.130:50627)
0806 02:55:36.138196 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:36.138216 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 02:55:36.138377 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:55:36.138688 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:55:36.138693 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:55:36.138717 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:55:36.139085 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:36.139095 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:36.140094 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:36.140099 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:36.141763 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:36.141777 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:36.142963 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:36.142971 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:36.143137 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:36.143163 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 02:55:36.143254 (+ 91us) client_negotiation.cc:240] Negotiation successful
0806 02:55:36.143386 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"mutex_wait_us":103,"thread_start_us":451,"threads_started":1}
I20260806 02:55:36.144062 16284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:36.137430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.131:39655 (local address 127.25.211.130:56723)
0806 02:55:36.138330 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:36.138345 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:55:36.138477 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:55:36.138794 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:55:36.138798 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:55:36.138839 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:55:36.139230 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:36.139240 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:36.140237 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:36.140242 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:36.141759 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:36.141774 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:36.143001 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:36.143008 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:36.143180 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:36.143203 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 02:55:36.143283 (+ 80us) client_negotiation.cc:240] Negotiation successful
0806 02:55:36.143410 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":783,"mutex_wait_us":26,"thread_start_us":442,"threads_started":1}
I20260806 02:55:36.144963 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e32d0cb55d354d37a1706a65ab759a7b" candidate_uuid: "0cb38587aca8444fb40d67e32941ea03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c5c116a405d4e698eb9b15b3c50a17e" is_pre_election: true
I20260806 02:55:36.145139 16180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e32d0cb55d354d37a1706a65ab759a7b" candidate_uuid: "0cb38587aca8444fb40d67e32941ea03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7acfb5f0395a47608eac1775aa86e11b" is_pre_election: true
I20260806 02:55:36.145551 15757 raft_consensus.cc:2454] T e32d0cb55d354d37a1706a65ab759a7b P 4c5c116a405d4e698eb9b15b3c50a17e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cb38587aca8444fb40d67e32941ea03 in term 0.
I20260806 02:55:36.145651 16180 raft_consensus.cc:2454] T e32d0cb55d354d37a1706a65ab759a7b P 7acfb5f0395a47608eac1775aa86e11b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cb38587aca8444fb40d67e32941ea03 in term 0.
I20260806 02:55:36.146304 15873 leader_election.cc:302] T e32d0cb55d354d37a1706a65ab759a7b P 0cb38587aca8444fb40d67e32941ea03 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cb38587aca8444fb40d67e32941ea03, 4c5c116a405d4e698eb9b15b3c50a17e; no voters:
I20260806 02:55:36.146744 16280 raft_consensus.cc:2781] T e32d0cb55d354d37a1706a65ab759a7b P 0cb38587aca8444fb40d67e32941ea03 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:55:36.146888 16280 raft_consensus.cc:482] T e32d0cb55d354d37a1706a65ab759a7b P 0cb38587aca8444fb40d67e32941ea03 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:55:36.146988 16280 raft_consensus.cc:3037] T e32d0cb55d354d37a1706a65ab759a7b P 0cb38587aca8444fb40d67e32941ea03 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:55:36.149276 16280 raft_consensus.cc:504] T e32d0cb55d354d37a1706a65ab759a7b P 0cb38587aca8444fb40d67e32941ea03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb38587aca8444fb40d67e32941ea03" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 32867 } } peers { permanent_uuid: "4c5c116a405d4e698eb9b15b3c50a17e" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 43767 } } peers { permanent_uuid: "7acfb5f0395a47608eac1775aa86e11b" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 39655 } }
I20260806 02:55:36.149909 16280 leader_election.cc:288] T e32d0cb55d354d37a1706a65ab759a7b P 0cb38587aca8444fb40d67e32941ea03 [CANDIDATE]: Term 1 election: Requested vote from peers 4c5c116a405d4e698eb9b15b3c50a17e (127.25.211.129:43767), 7acfb5f0395a47608eac1775aa86e11b (127.25.211.131:39655)
I20260806 02:55:36.150497 16180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e32d0cb55d354d37a1706a65ab759a7b" candidate_uuid: "0cb38587aca8444fb40d67e32941ea03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7acfb5f0395a47608eac1775aa86e11b"
I20260806 02:55:36.150545 15757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e32d0cb55d354d37a1706a65ab759a7b" candidate_uuid: "0cb38587aca8444fb40d67e32941ea03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c5c116a405d4e698eb9b15b3c50a17e"
I20260806 02:55:36.150701 16180 raft_consensus.cc:3037] T e32d0cb55d354d37a1706a65ab759a7b P 7acfb5f0395a47608eac1775aa86e11b [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:55:36.150743 15757 raft_consensus.cc:3037] T e32d0cb55d354d37a1706a65ab759a7b P 4c5c116a405d4e698eb9b15b3c50a17e [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:55:36.152966 16180 raft_consensus.cc:2454] T e32d0cb55d354d37a1706a65ab759a7b P 7acfb5f0395a47608eac1775aa86e11b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cb38587aca8444fb40d67e32941ea03 in term 1.
I20260806 02:55:36.152976 15757 raft_consensus.cc:2454] T e32d0cb55d354d37a1706a65ab759a7b P 4c5c116a405d4e698eb9b15b3c50a17e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cb38587aca8444fb40d67e32941ea03 in term 1.
I20260806 02:55:36.153533 15879 leader_election.cc:302] T e32d0cb55d354d37a1706a65ab759a7b P 0cb38587aca8444fb40d67e32941ea03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cb38587aca8444fb40d67e32941ea03, 7acfb5f0395a47608eac1775aa86e11b; no voters:
I20260806 02:55:36.153867 16280 raft_consensus.cc:2781] T e32d0cb55d354d37a1706a65ab759a7b P 0cb38587aca8444fb40d67e32941ea03 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:55:36.154372 16280 raft_consensus.cc:686] T e32d0cb55d354d37a1706a65ab759a7b P 0cb38587aca8444fb40d67e32941ea03 [term 1 LEADER]: Becoming Leader. State: Replica: 0cb38587aca8444fb40d67e32941ea03, State: Running, Role: LEADER
I20260806 02:55:36.154879 16280 consensus_queue.cc:237] T e32d0cb55d354d37a1706a65ab759a7b P 0cb38587aca8444fb40d67e32941ea03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb38587aca8444fb40d67e32941ea03" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 32867 } } peers { permanent_uuid: "4c5c116a405d4e698eb9b15b3c50a17e" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 43767 } } peers { permanent_uuid: "7acfb5f0395a47608eac1775aa86e11b" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 39655 } }
I20260806 02:55:36.159713 15512 catalog_manager.cc:5495] T e32d0cb55d354d37a1706a65ab759a7b P 0cb38587aca8444fb40d67e32941ea03 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0cb38587aca8444fb40d67e32941ea03 (127.25.211.130). New cstate: current_term: 1 leader_uuid: "0cb38587aca8444fb40d67e32941ea03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb38587aca8444fb40d67e32941ea03" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 32867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c5c116a405d4e698eb9b15b3c50a17e" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 43767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7acfb5f0395a47608eac1775aa86e11b" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 39655 } health_report { overall_health: UNKNOWN } } }
I20260806 02:55:36.200860 16259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:36.195934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35934 (local address 127.25.211.130:32867)
0806 02:55:36.196119 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:36.196123 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:36.196140 (+ 17us) server_negotiation.cc:407] Connection header received
0806 02:55:36.196227 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:36.196231 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:36.196318 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:36.196436 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:55:36.196736 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:36.197432 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:36.198844 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:36.199290 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:36.199384 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:36.199852 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:55:36.200381 (+ 529us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:55:36.200486 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:36.200562 (+ 76us) server_negotiation.cc:299] Negotiation successful
0806 02:55:36.200633 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260806 02:55:36.205310 16241 tablet.cc:2369] T e32d0cb55d354d37a1706a65ab759a7b P 7acfb5f0395a47608eac1775aa86e11b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:55:36.209990 16180 raft_consensus.cc:1261] T e32d0cb55d354d37a1706a65ab759a7b P 7acfb5f0395a47608eac1775aa86e11b [term 1 FOLLOWER]: Refusing update from remote peer 0cb38587aca8444fb40d67e32941ea03: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:55:36.209990 15757 raft_consensus.cc:1261] T e32d0cb55d354d37a1706a65ab759a7b P 4c5c116a405d4e698eb9b15b3c50a17e [term 1 FOLLOWER]: Refusing update from remote peer 0cb38587aca8444fb40d67e32941ea03: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:55:36.211046 16285 consensus_queue.cc:1035] T e32d0cb55d354d37a1706a65ab759a7b P 0cb38587aca8444fb40d67e32941ea03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c5c116a405d4e698eb9b15b3c50a17e" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 43767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:55:36.211422 16280 consensus_queue.cc:1035] T e32d0cb55d354d37a1706a65ab759a7b P 0cb38587aca8444fb40d67e32941ea03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7acfb5f0395a47608eac1775aa86e11b" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 39655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 02:55:36.221338 16053 tablet.cc:2369] T e32d0cb55d354d37a1706a65ab759a7b P 0cb38587aca8444fb40d67e32941ea03: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:55:36.226188 15841 tablet.cc:2369] T e32d0cb55d354d37a1706a65ab759a7b P 4c5c116a405d4e698eb9b15b3c50a17e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:55:36.229003 16292 mvcc.cc:205] Tried to move back new op lower bound from 7315394298705031168 to 7315394298494414848. Current Snapshot: MvccSnapshot[applied={T|T < 7315394298705031168}]
I20260806 02:55:36.232635 16290 mvcc.cc:205] Tried to move back new op lower bound from 7315394298705031168 to 7315394298494414848. Current Snapshot: MvccSnapshot[applied={T|T < 7315394298705031168}]
I20260806 02:55:36.239373 15512 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:43104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:55:36.239651 15512 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:43104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:55:36.243197 15512 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4: Sending DeleteTablet for 3 replicas of tablet e32d0cb55d354d37a1706a65ab759a7b
I20260806 02:55:36.244565 15949 tablet_service.cc:1514] Processing DeleteTablet for tablet e32d0cb55d354d37a1706a65ab759a7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:55:36 UTC) from {username='kudu', principal='kudu/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:35930
I20260806 02:55:36.244578 15718 tablet_service.cc:1514] Processing DeleteTablet for tablet e32d0cb55d354d37a1706a65ab759a7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:55:36 UTC) from {username='kudu', principal='kudu/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:60110
I20260806 02:55:36.244714 16148 tablet_service.cc:1514] Processing DeleteTablet for tablet e32d0cb55d354d37a1706a65ab759a7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:55:36 UTC) from {username='kudu', principal='kudu/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:53780
I20260806 02:55:36.246405 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 15575
I20260806 02:55:36.259883 16299 tablet_replica.cc:330] T e32d0cb55d354d37a1706a65ab759a7b P 0cb38587aca8444fb40d67e32941ea03: stopping tablet replica
I20260806 02:55:36.260474 16299 raft_consensus.cc:2229] T e32d0cb55d354d37a1706a65ab759a7b P 0cb38587aca8444fb40d67e32941ea03 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:55:36.261236 16299 raft_consensus.cc:2258] T e32d0cb55d354d37a1706a65ab759a7b P 0cb38587aca8444fb40d67e32941ea03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:55:36.264922 16301 tablet_replica.cc:330] T e32d0cb55d354d37a1706a65ab759a7b P 7acfb5f0395a47608eac1775aa86e11b: stopping tablet replica
I20260806 02:55:36.265443 16301 raft_consensus.cc:2229] T e32d0cb55d354d37a1706a65ab759a7b P 7acfb5f0395a47608eac1775aa86e11b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:55:36.266093 16301 raft_consensus.cc:2258] T e32d0cb55d354d37a1706a65ab759a7b P 7acfb5f0395a47608eac1775aa86e11b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:55:36.266600 16261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:36.247440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60124 (local address 127.25.211.129:43767)
0806 02:55:36.247637 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:36.247644 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:36.247669 (+ 25us) server_negotiation.cc:407] Connection header received
0806 02:55:36.247778 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:36.247784 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:36.247871 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:36.247992 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:36.249811 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:36.251154 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:36.253180 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:36.253894 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:36.254043 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:36.257951 (+ 3908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:36.257976 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:36.258004 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:36.258079 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:36.262626 (+ 4547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:36.263724 (+ 1098us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:36.263735 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:36.263739 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:36.263967 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:36.264409 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:36.264413 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:36.264417 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:36.264943 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:36.265378 (+ 435us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:36.265858 (+ 480us) server_negotiation.cc:299] Negotiation successful
0806 02:55:36.266131 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 02:55:36.267244 16300 tablet_replica.cc:330] T e32d0cb55d354d37a1706a65ab759a7b P 4c5c116a405d4e698eb9b15b3c50a17e: stopping tablet replica
I20260806 02:55:36.267606 15830 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:55:36.268180 16299 ts_tablet_manager.cc:1905] T e32d0cb55d354d37a1706a65ab759a7b P 0cb38587aca8444fb40d67e32941ea03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:55:36.269165 16301 ts_tablet_manager.cc:1905] T e32d0cb55d354d37a1706a65ab759a7b P 7acfb5f0395a47608eac1775aa86e11b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:55:36.276000 16299 ts_tablet_manager.cc:1918] T e32d0cb55d354d37a1706a65ab759a7b P 0cb38587aca8444fb40d67e32941ea03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:55:36.276224 16299 log.cc:1196] T e32d0cb55d354d37a1706a65ab759a7b P 0cb38587aca8444fb40d67e32941ea03: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-1/wal/wals/e32d0cb55d354d37a1706a65ab759a7b
I20260806 02:55:36.276854 16299 ts_tablet_manager.cc:1939] T e32d0cb55d354d37a1706a65ab759a7b P 0cb38587aca8444fb40d67e32941ea03: Deleting consensus metadata
I20260806 02:55:36.276849 16301 ts_tablet_manager.cc:1918] T e32d0cb55d354d37a1706a65ab759a7b P 7acfb5f0395a47608eac1775aa86e11b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:55:36.277014 16301 log.cc:1196] T e32d0cb55d354d37a1706a65ab759a7b P 7acfb5f0395a47608eac1775aa86e11b: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-2/wal/wals/e32d0cb55d354d37a1706a65ab759a7b
I20260806 02:55:36.277611 16301 ts_tablet_manager.cc:1939] T e32d0cb55d354d37a1706a65ab759a7b P 7acfb5f0395a47608eac1775aa86e11b: Deleting consensus metadata
I20260806 02:55:36.279013 15496 catalog_manager.cc:4841] TS 0cb38587aca8444fb40d67e32941ea03 (127.25.211.130:32867): tablet e32d0cb55d354d37a1706a65ab759a7b (table test-table [id=6d12b9c82ddf427691ff81eeeabfef1b]) successfully deleted
I20260806 02:55:36.279606 15497 catalog_manager.cc:4841] TS 7acfb5f0395a47608eac1775aa86e11b (127.25.211.131:39655): tablet e32d0cb55d354d37a1706a65ab759a7b (table test-table [id=6d12b9c82ddf427691ff81eeeabfef1b]) successfully deleted
I20260806 02:55:36.267699 16300 raft_consensus.cc:2229] T e32d0cb55d354d37a1706a65ab759a7b P 4c5c116a405d4e698eb9b15b3c50a17e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:55:36.369410 16300 raft_consensus.cc:2258] T e32d0cb55d354d37a1706a65ab759a7b P 4c5c116a405d4e698eb9b15b3c50a17e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:55:36.370085 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 15575
W20260806 02:55:36.397473 15495 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.211.129:43767 (error 108)
I20260806 02:55:36.398249 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 15847
I20260806 02:55:36.399562 16262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:36.399002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:43767 (local address 127.0.0.1:60136)
0806 02:55:36.399279 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:36.399394 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.129:43767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
W20260806 02:55:36.399801 15495 catalog_manager.cc:4584] TS 4c5c116a405d4e698eb9b15b3c50a17e (127.25.211.129:43767): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e32d0cb55d354d37a1706a65ab759a7b: Network error: Client connection negotiation failed: client connection to 127.25.211.129:43767: connect: Connection refused (error 111)
I20260806 02:55:36.413902 16259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:36.399072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35948 (local address 127.25.211.130:32867)
0806 02:55:36.399255 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:36.399265 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:36.399404 (+ 139us) server_negotiation.cc:407] Connection header received
0806 02:55:36.399585 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:36.399589 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:36.399687 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:36.399796 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:36.401228 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:36.402413 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:36.404294 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:36.404748 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:36.404869 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:36.407925 (+ 3056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:36.407937 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:36.407954 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:36.407995 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:36.411049 (+ 3054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:36.411749 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:36.411757 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:36.411760 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:36.411908 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:36.412302 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:36.412306 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:36.412308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:36.412683 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:36.413022 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:36.413310 (+ 288us) server_negotiation.cc:299] Negotiation successful
0806 02:55:36.413540 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 02:55:36.414651 16045 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:55:36.453439 16262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:36.452959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:43767 (local address 127.0.0.1:60150)
0806 02:55:36.453174 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:36.453275 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.129:43767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260806 02:55:36.454556 16262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:36.454188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:43767 (local address 127.0.0.1:60154)
0806 02:55:36.454336 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:36.454425 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.129:43767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260806 02:55:36.515245 16262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:36.514723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:43767 (local address 127.0.0.1:60156)
0806 02:55:36.514930 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:36.515045 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.129:43767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260806 02:55:36.516363 16262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:36.515996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:43767 (local address 127.0.0.1:60158)
0806 02:55:36.516157 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:36.516239 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.129:43767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260806 02:55:36.540134 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 15847
I20260806 02:55:36.564699 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 16055
I20260806 02:55:36.578882 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:36.565495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53786 (local address 127.25.211.131:39655)
0806 02:55:36.565685 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:36.565691 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:36.565712 (+ 21us) server_negotiation.cc:407] Connection header received
0806 02:55:36.565910 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:36.565914 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:36.565995 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:36.566082 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:36.567541 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:36.568342 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:36.569715 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:36.570124 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:36.570496 (+ 372us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:36.573110 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:36.573125 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:36.573143 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:36.573179 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:36.575986 (+ 2807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:36.576738 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:36.576746 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:36.576748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:36.576911 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:36.577283 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:36.577286 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:36.577289 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:36.577670 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:36.577984 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:36.578329 (+ 345us) server_negotiation.cc:299] Negotiation successful
0806 02:55:36.578561 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 02:55:36.579582 16233 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:55:36.597674 16262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:36.597196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:43767 (local address 127.0.0.1:60160)
0806 02:55:36.597417 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:36.597508 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.129:43767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260806 02:55:36.598837 16262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:36.598445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:43767 (local address 127.0.0.1:60174)
0806 02:55:36.598638 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:36.598713 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.129:43767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260806 02:55:36.669727 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 16055
I20260806 02:55:36.692586 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 15470
I20260806 02:55:36.693835 15564 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:55:36.809477 16262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:36.808725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:43767 (local address 127.0.0.1:60180)
0806 02:55:36.808944 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:36.809172 (+ 228us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.129:43767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260806 02:55:36.809613 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 15470
I20260806 02:55:36.838307 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:45343
--webserver_interface=127.25.211.190
--webserver_port=34747
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:36275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.211.190:45343
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:55:37.182971 16312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:55:37.183293 16312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:55:37.183436 16312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:55:37.190868 16312 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:55:37.191015 16312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:55:37.191105 16312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:55:37.191188 16312 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:55:37.191250 16312 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:55:37.201421 16312 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:36275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.211.190:45343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:45343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.211.190
--webserver_port=34747
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:55:37.202867 16312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:55:37.204221 16312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:55:37.221139 16320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:37.221654 16325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:37.222011 16322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:55:37.223434 16312 server_base.cc:1034] running on GCE node
I20260806 02:55:37.224354 16312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:55:37.226054 16312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:55:37.227269 16312 hybrid_clock.cc:648] HybridClock initialized: now 1785984937227219 us; error 43 us; skew 500 ppm
I20260806 02:55:37.227644 16312 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:55:37 dist-test-slave-zj8q krb5kdc[14605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984937, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:55:37.230969 16312 init.cc:377] Logged in from keytab as kudu/127.25.211.190@KRBTEST.COM (short username kudu)
I20260806 02:55:37.232503 16312 webserver.cc:460] Webserver started at http://127.25.211.190:34747/ using document root <none> and password file <none>
I20260806 02:55:37.233197 16312 fs_manager.cc:359] Metadata directory not provided
I20260806 02:55:37.233304 16312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:55:37.239135 16312 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 02:55:37.242192 16336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:37.243562 16312 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 02:55:37.243747 16312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/master-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/master-0/wal
uuid: "7b628097688e47b7a0187a2f2f3a54e4"
format_stamp: "Formatted at 2026-08-06 02:55:31 on dist-test-slave-zj8q"
server_key: "4ef30fd60b93baa3a772901be83b1b44"
server_key_iv: "f0d33b683b62622baae7293d27b57725"
server_key_version: "encryptionkey@0"
I20260806 02:55:37.244778 16312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:55:37.261195 16312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:55:37.265349 16312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:55:37.265628 16312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:55:37.283383 16312 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.190:45343
I20260806 02:55:37.283403 16405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.190:45343 every 8 connection(s)
I20260806 02:55:37.284945 16312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
I20260806 02:55:37.287633 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 16312
I20260806 02:55:37.288650 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.129:43767
--local_ip_for_outbound_sockets=127.25.211.129
--tserver_master_addrs=127.25.211.190:45343
--webserver_port=41871
--webserver_interface=127.25.211.129
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab
--principal=kudu/127.25.211.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:36275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:55:37.294453 16406 sys_catalog.cc:263] Verifying existing consensus state
I20260806 02:55:37.299237 16406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4: Bootstrap starting.
I20260806 02:55:37.320564 16406 log.cc:824] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4: Log is configured to *not* fsync() on all Append() calls
I20260806 02:55:37.337915 16406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:55:37.338564 16406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4: Bootstrap complete.
I20260806 02:55:37.348143 16406 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b628097688e47b7a0187a2f2f3a54e4" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45343 } }
I20260806 02:55:37.349231 16406 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b628097688e47b7a0187a2f2f3a54e4, State: Initialized, Role: FOLLOWER
I20260806 02:55:37.349918 16406 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b628097688e47b7a0187a2f2f3a54e4" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45343 } }
I20260806 02:55:37.350178 16406 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:55:37.350314 16406 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:55:37.350483 16406 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4 [term 2 FOLLOWER]: Advancing to term 3
I20260806 02:55:37.354882 16406 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b628097688e47b7a0187a2f2f3a54e4" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45343 } }
I20260806 02:55:37.355460 16406 leader_election.cc:302] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b628097688e47b7a0187a2f2f3a54e4; no voters:
I20260806 02:55:37.356117 16406 leader_election.cc:288] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260806 02:55:37.356266 16413 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 02:55:37.358903 16413 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4 [term 3 LEADER]: Becoming Leader. State: Replica: 7b628097688e47b7a0187a2f2f3a54e4, State: Running, Role: LEADER
I20260806 02:55:37.359462 16413 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b628097688e47b7a0187a2f2f3a54e4" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45343 } }
I20260806 02:55:37.359974 16406 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:55:37.361804 16418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b628097688e47b7a0187a2f2f3a54e4. Latest consensus state: current_term: 3 leader_uuid: "7b628097688e47b7a0187a2f2f3a54e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b628097688e47b7a0187a2f2f3a54e4" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45343 } } }
I20260806 02:55:37.362323 16415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "7b628097688e47b7a0187a2f2f3a54e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b628097688e47b7a0187a2f2f3a54e4" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45343 } } }
I20260806 02:55:37.362589 16415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4 [sys.catalog]: This master's current role is: LEADER
I20260806 02:55:37.363391 16418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b628097688e47b7a0187a2f2f3a54e4 [sys.catalog]: This master's current role is: LEADER
I20260806 02:55:37.366731 16424 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:55:37.375370 16424 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:55:37.376925 16424 catalog_manager.cc:1259] Loaded cluster ID: a5f8fb30977e47c9b2037092c1b5bb2d
I20260806 02:55:37.377038 16424 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260806 02:55:37.379141 16424 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fd0381919ad google::LogMessage::Fail() at ??:0
@ 0x7fd03819488a google::LogMessage::SendToLog() at ??:0
@ 0x7fd0381914de google::LogMessage::Flush() at ??:0
@ 0x7fd038192a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fd04dbc4512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fd04dbc1976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fd04dc49610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fd04dc49439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fd03addd345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fd03ade091c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fd03ade0799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fd03adbbeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd03e0956db start_thread at ??:0
@ 0x7fd03702b61f clone at ??:0
W20260806 02:55:37.675041 16409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:55:37.675410 16409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:55:37.675563 16409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:55:37.683199 16409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:55:37.683337 16409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:55:37.683496 16409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.129
I20260806 02:55:37.693703 16409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:36275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab
--principal=kudu/127.25.211.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.129:43767
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.211.129
--webserver_port=41871
--tserver_master_addrs=127.25.211.190:45343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.129
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:55:37.695145 16409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:55:37.696534 16409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:55:37.710368 16480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:37.710287 16482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:37.710340 16485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:55:37.711329 16409 server_base.cc:1034] running on GCE node
I20260806 02:55:37.712024 16409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:55:37.713070 16409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:55:37.714293 16409 hybrid_clock.cc:648] HybridClock initialized: now 1785984937714218 us; error 61 us; skew 500 ppm
I20260806 02:55:37.714646 16409 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:55:37 dist-test-slave-zj8q krb5kdc[14605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984937, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:55:37.717952 16409 init.cc:377] Logged in from keytab as kudu/127.25.211.129@KRBTEST.COM (short username kudu)
I20260806 02:55:37.719487 16409 webserver.cc:460] Webserver started at http://127.25.211.129:41871/ using document root <none> and password file <none>
I20260806 02:55:37.720225 16409 fs_manager.cc:359] Metadata directory not provided
I20260806 02:55:37.720345 16409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:55:37.726197 16409 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260806 02:55:37.731640 16494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:37.733206 16409 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:55:37.733392 16409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-0/wal
uuid: "4c5c116a405d4e698eb9b15b3c50a17e"
format_stamp: "Formatted at 2026-08-06 02:55:31 on dist-test-slave-zj8q"
server_key: "3aa6a7f11d98841dc43f0e707e8bc873"
server_key_iv: "fce77d93f36ef050aebc4890ec034c96"
server_key_version: "encryptionkey@0"
I20260806 02:55:37.734512 16409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:55:37.754279 16409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:55:37.758780 16409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:55:37.759056 16409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:55:37.760108 16409 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:55:37.762238 16503 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260806 02:55:37.768388 16409 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 02:55:37.768538 16409 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260806 02:55:37.768731 16409 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 02:55:37.772163 16409 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 02:55:37.772269 16409 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.002s sys 0.000s
I20260806 02:55:37.772720 16503 tablet_bootstrap.cc:492] T e32d0cb55d354d37a1706a65ab759a7b P 4c5c116a405d4e698eb9b15b3c50a17e: Bootstrap starting.
I20260806 02:55:37.781632 16503 log.cc:824] T e32d0cb55d354d37a1706a65ab759a7b P 4c5c116a405d4e698eb9b15b3c50a17e: Log is configured to *not* fsync() on all Append() calls
I20260806 02:55:37.799536 16503 tablet_bootstrap.cc:492] T e32d0cb55d354d37a1706a65ab759a7b P 4c5c116a405d4e698eb9b15b3c50a17e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:55:37.800714 16503 tablet_bootstrap.cc:492] T e32d0cb55d354d37a1706a65ab759a7b P 4c5c116a405d4e698eb9b15b3c50a17e: Bootstrap complete.
I20260806 02:55:37.801424 16503 ts_tablet_manager.cc:1397] T e32d0cb55d354d37a1706a65ab759a7b P 4c5c116a405d4e698eb9b15b3c50a17e: Time spent bootstrapping tablet: real 0.029s user 0.011s sys 0.014s
I20260806 02:55:37.809914 16409 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.129:43767
I20260806 02:55:37.809955 16670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.129:43767 every 8 connection(s)
I20260806 02:55:37.811770 16409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
I20260806 02:55:37.813392 16503 raft_consensus.cc:348] T e32d0cb55d354d37a1706a65ab759a7b P 4c5c116a405d4e698eb9b15b3c50a17e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb38587aca8444fb40d67e32941ea03" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 32867 } } peers { permanent_uuid: "4c5c116a405d4e698eb9b15b3c50a17e" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 43767 } } peers { permanent_uuid: "7acfb5f0395a47608eac1775aa86e11b" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 39655 } }
I20260806 02:55:37.814680 16503 raft_consensus.cc:729] T e32d0cb55d354d37a1706a65ab759a7b P 4c5c116a405d4e698eb9b15b3c50a17e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c5c116a405d4e698eb9b15b3c50a17e, State: Initialized, Role: FOLLOWER
I20260806 02:55:37.815522 16503 consensus_queue.cc:260] T e32d0cb55d354d37a1706a65ab759a7b P 4c5c116a405d4e698eb9b15b3c50a17e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb38587aca8444fb40d67e32941ea03" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 32867 } } peers { permanent_uuid: "4c5c116a405d4e698eb9b15b3c50a17e" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 43767 } } peers { permanent_uuid: "7acfb5f0395a47608eac1775aa86e11b" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 39655 } }
I20260806 02:55:37.817320 16503 ts_tablet_manager.cc:1428] T e32d0cb55d354d37a1706a65ab759a7b P 4c5c116a405d4e698eb9b15b3c50a17e: Time spent starting tablet: real 0.016s user 0.009s sys 0.009s
I20260806 02:55:37.817739 16673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:37.815267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45343 (local address 127.25.211.129:52329)
0806 02:55:37.816428 (+ 1161us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:37.816769 (+ 341us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":782,"thread_start_us":521,"threads_started":1}
W20260806 02:55:37.818437 16671 heartbeater.cc:644] Failed to heartbeat to 127.25.211.190:45343 (0 consecutive failures): Network error: Failed to ping master at 127.25.211.190:45343: Client connection negotiation failed: client connection to 127.25.211.190:45343: connect: Connection refused (error 111)
I20260806 02:55:37.819317 16673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:37.818985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45343 (local address 127.25.211.129:38087)
0806 02:55:37.819138 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:37.819208 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260806 02:55:37.820155 16673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:37.819822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45343 (local address 127.25.211.129:48279)
0806 02:55:37.819954 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:37.820030 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260806 02:55:37.820403 16671 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 02:55:37.822074 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 16409
I20260806 02:55:37.822751 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.130:32867
--local_ip_for_outbound_sockets=127.25.211.130
--tserver_master_addrs=127.25.211.190:45343
--webserver_port=37671
--webserver_interface=127.25.211.130
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab
--principal=kudu/127.25.211.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:36275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:55:38.170115 16677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:55:38.170447 16677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:55:38.170572 16677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:55:38.177873 16677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:55:38.178004 16677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:55:38.178177 16677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.130
I20260806 02:55:38.188263 16677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:36275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab
--principal=kudu/127.25.211.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.130:32867
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.211.130
--webserver_port=37671
--tserver_master_addrs=127.25.211.190:45343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.130
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:55:38.189671 16677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:55:38.191113 16677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:55:38.204592 16682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:38.204640 16684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:38.204566 16687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:55:38.205245 16677 server_base.cc:1034] running on GCE node
I20260806 02:55:38.205967 16677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:55:38.207019 16677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:55:38.208240 16677 hybrid_clock.cc:648] HybridClock initialized: now 1785984938208172 us; error 62 us; skew 500 ppm
I20260806 02:55:38.208585 16677 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:55:38 dist-test-slave-zj8q krb5kdc[14605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984938, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:55:38.211973 16677 init.cc:377] Logged in from keytab as kudu/127.25.211.130@KRBTEST.COM (short username kudu)
I20260806 02:55:38.213639 16677 webserver.cc:460] Webserver started at http://127.25.211.130:37671/ using document root <none> and password file <none>
I20260806 02:55:38.214428 16677 fs_manager.cc:359] Metadata directory not provided
I20260806 02:55:38.214550 16677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:55:38.220539 16677 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:55:38.223719 16697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:38.225288 16677 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:55:38.225484 16677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-1/wal
uuid: "0cb38587aca8444fb40d67e32941ea03"
format_stamp: "Formatted at 2026-08-06 02:55:32 on dist-test-slave-zj8q"
server_key: "98e38b2e4a4e0b25bca5309f3b32bde0"
server_key_iv: "1384851389e32776e8555760b7c71c14"
server_key_version: "encryptionkey@0"
I20260806 02:55:38.226639 16677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:55:38.243876 16677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:55:38.248660 16677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:55:38.248950 16677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:55:38.250068 16677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:55:38.251597 16677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:55:38.251696 16677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:38.251807 16677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:55:38.251883 16677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:38.284406 16677 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.130:32867
I20260806 02:55:38.284464 16858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.130:32867 every 8 connection(s)
I20260806 02:55:38.285920 16677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
I20260806 02:55:38.290788 16863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:38.288929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45343 (local address 127.25.211.130:43805)
0806 02:55:38.289687 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:38.289919 (+ 232us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":298,"threads_started":1}
W20260806 02:55:38.291327 16861 heartbeater.cc:644] Failed to heartbeat to 127.25.211.190:45343 (0 consecutive failures): Network error: Failed to ping master at 127.25.211.190:45343: Client connection negotiation failed: client connection to 127.25.211.190:45343: connect: Connection refused (error 111)
I20260806 02:55:38.292073 16863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:38.291789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45343 (local address 127.25.211.130:32805)
0806 02:55:38.291914 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:38.291977 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260806 02:55:38.292770 16863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:38.292502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45343 (local address 127.25.211.130:53351)
0806 02:55:38.292601 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:38.292667 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260806 02:55:38.292994 16861 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 02:55:38.293887 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 16677
I20260806 02:55:38.294521 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.131:39655
--local_ip_for_outbound_sockets=127.25.211.131
--tserver_master_addrs=127.25.211.190:45343
--webserver_port=42341
--webserver_interface=127.25.211.131
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab
--principal=kudu/127.25.211.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:36275
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:55:38.633822 16864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:55:38.634116 16864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:55:38.634267 16864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:55:38.641601 16864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:55:38.641726 16864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:55:38.641881 16864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.131
I20260806 02:55:38.651825 16864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:36275
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab
--principal=kudu/127.25.211.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.131:39655
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.211.131
--webserver_port=42341
--tserver_master_addrs=127.25.211.190:45343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.131
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:55:38.653209 16864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:55:38.654596 16864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:55:38.667760 16875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:38.667812 16876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:38.667867 16879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:55:38.668972 16864 server_base.cc:1034] running on GCE node
I20260806 02:55:38.669626 16864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:55:38.670660 16864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:55:38.671852 16864 hybrid_clock.cc:648] HybridClock initialized: now 1785984938671791 us; error 57 us; skew 500 ppm
I20260806 02:55:38.672164 16864 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:55:38 dist-test-slave-zj8q krb5kdc[14605](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984938, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:55:38.675415 16864 init.cc:377] Logged in from keytab as kudu/127.25.211.131@KRBTEST.COM (short username kudu)
I20260806 02:55:38.676923 16864 webserver.cc:460] Webserver started at http://127.25.211.131:42341/ using document root <none> and password file <none>
I20260806 02:55:38.677695 16864 fs_manager.cc:359] Metadata directory not provided
I20260806 02:55:38.677840 16864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:55:38.683483 16864 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 02:55:38.686513 16890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:38.687960 16864 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260806 02:55:38.688139 16864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-2/wal
uuid: "7acfb5f0395a47608eac1775aa86e11b"
format_stamp: "Formatted at 2026-08-06 02:55:33 on dist-test-slave-zj8q"
server_key: "9aa9fb40458463c7dd8c14404dab453e"
server_key_iv: "cb04e582455cef0709d5070676529619"
server_key_version: "encryptionkey@0"
I20260806 02:55:38.689134 16864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:55:38.707659 16864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:55:38.711959 16864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:55:38.712220 16864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:55:38.713224 16864 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:55:38.714582 16864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:55:38.714679 16864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:38.714797 16864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:55:38.714869 16864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:38.746402 16864 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.131:39655
I20260806 02:55:38.746464 17057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.131:39655 every 8 connection(s)
I20260806 02:55:38.747888 16864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
I20260806 02:55:38.752873 17062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:38.750948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45343 (local address 127.25.211.131:54873)
0806 02:55:38.751767 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:38.751990 (+ 223us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":307,"threads_started":1}
W20260806 02:55:38.753687 17058 heartbeater.cc:644] Failed to heartbeat to 127.25.211.190:45343 (0 consecutive failures): Network error: Failed to ping master at 127.25.211.190:45343: Client connection negotiation failed: client connection to 127.25.211.190:45343: connect: Connection refused (error 111)
I20260806 02:55:38.753691 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 16864
I20260806 02:55:38.754742 17062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:38.754288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45343 (local address 127.25.211.131:58033)
0806 02:55:38.754559 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:38.754626 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260806 02:55:38.755582 17062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:38.755268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45343 (local address 127.25.211.131:35585)
0806 02:55:38.755401 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:38.755477 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260806 02:55:38.755792 17058 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 02:55:38.821872 17064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:38.821033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45343 (local address 127.25.211.129:41969)
0806 02:55:38.821552 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:38.821653 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":254,"threads_started":1}
I20260806 02:55:39.005761 17066 raft_consensus.cc:482] T e32d0cb55d354d37a1706a65ab759a7b P 4c5c116a405d4e698eb9b15b3c50a17e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:55:39.006065 17066 raft_consensus.cc:504] T e32d0cb55d354d37a1706a65ab759a7b P 4c5c116a405d4e698eb9b15b3c50a17e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb38587aca8444fb40d67e32941ea03" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 32867 } } peers { permanent_uuid: "4c5c116a405d4e698eb9b15b3c50a17e" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 43767 } } peers { permanent_uuid: "7acfb5f0395a47608eac1775aa86e11b" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 39655 } }
I20260806 02:55:39.007793 17066 leader_election.cc:288] T e32d0cb55d354d37a1706a65ab759a7b P 4c5c116a405d4e698eb9b15b3c50a17e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0cb38587aca8444fb40d67e32941ea03 (127.25.211.130:32867), 7acfb5f0395a47608eac1775aa86e11b (127.25.211.131:39655)
Aug 06 02:55:39 dist-test-slave-zj8q krb5kdc[14605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984937, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.129@KRBTEST.COM for kudu/127.25.211.131@KRBTEST.COM
Aug 06 02:55:39 dist-test-slave-zj8q krb5kdc[14605](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984937, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.129@KRBTEST.COM for kudu/127.25.211.130@KRBTEST.COM
I20260806 02:55:39.030701 17068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:39.008365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:60011 (local address 127.25.211.130:32867)
0806 02:55:39.010572 (+ 2207us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:39.010582 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:39.010613 (+ 31us) server_negotiation.cc:407] Connection header received
0806 02:55:39.012034 (+ 1421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:39.012044 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:39.012426 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:39.012803 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:39.014849 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:39.016145 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:39.017949 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:39.018650 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:39.018838 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:39.023222 (+ 4384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:39.023230 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:39.023244 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:39.023286 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:39.026505 (+ 3219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:39.027429 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:39.027440 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:39.027446 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:39.027632 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:39.028087 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:39.028093 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:39.028096 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:39.028736 (+ 640us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:39.028922 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:39.029808 (+ 886us) server_negotiation.cc:299] Negotiation successful
0806 02:55:39.030253 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2065,"thread_start_us":1948,"threads_started":1}
I20260806 02:55:39.030692 17064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:39.008048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.130:32867 (local address 127.25.211.129:60011)
0806 02:55:39.008415 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:39.008455 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 02:55:39.009113 (+ 658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:55:39.012981 (+ 3868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:55:39.012992 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:55:39.013766 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:55:39.014576 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:39.014607 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:39.016355 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:39.016360 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:39.017793 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:39.017807 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:39.018864 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:39.018871 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:39.019110 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:39.019895 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:55:39.019912 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:55:39.022920 (+ 3008us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:55:39.026666 (+ 3746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:39.026672 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:39.026684 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:39.027282 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:39.027760 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:39.027763 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:39.027766 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:39.027972 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:39.028892 (+ 920us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:55:39.028900 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:55:39.029233 (+ 333us) client_negotiation.cc:769] Sending connection context
0806 02:55:39.029545 (+ 312us) client_negotiation.cc:240] Negotiation successful
0806 02:55:39.030198 (+ 653us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":87,"spinlock_wait_cycles":36992}
I20260806 02:55:39.030843 17067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:39.008171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.131:39655 (local address 127.25.211.129:36959)
0806 02:55:39.009933 (+ 1762us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:39.009952 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:55:39.010069 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:55:39.012981 (+ 2912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:55:39.012992 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:55:39.013782 (+ 790us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:55:39.014585 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:39.014609 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:39.016351 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:39.016356 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:39.017790 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:39.017806 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:39.018913 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:39.018922 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:39.019111 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:39.019855 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:55:39.019888 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:55:39.022673 (+ 2785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:55:39.026648 (+ 3975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:39.026659 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:39.026676 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:39.027285 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:39.027766 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:39.027769 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:39.027772 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:39.027977 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:39.028892 (+ 915us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:55:39.028900 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:55:39.029233 (+ 333us) client_negotiation.cc:769] Sending connection context
0806 02:55:39.029545 (+ 312us) client_negotiation.cc:240] Negotiation successful
0806 02:55:39.030369 (+ 824us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1618,"mutex_wait_us":425,"spinlock_wait_cycles":30080,"thread_start_us":1431,"threads_started":1}
I20260806 02:55:39.030794 17069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:39.008565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:36959 (local address 127.25.211.131:39655)
0806 02:55:39.010471 (+ 1906us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:39.010483 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:39.010521 (+ 38us) server_negotiation.cc:407] Connection header received
0806 02:55:39.012035 (+ 1514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:39.012044 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:39.012426 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:39.012740 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:39.014848 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:39.016145 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:39.017951 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:39.018663 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:39.018837 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:39.023013 (+ 4176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:39.023022 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:39.023040 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:39.023088 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:39.026462 (+ 3374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:39.027436 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:39.027444 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:39.027449 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:39.027639 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:39.028088 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:39.028094 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:39.028097 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:39.028736 (+ 639us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:39.028921 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:39.029873 (+ 952us) server_negotiation.cc:299] Negotiation successful
0806 02:55:39.030293 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1714,"thread_start_us":1425,"threads_started":1}
W20260806 02:55:39.034617 16498 leader_election.cc:341] T e32d0cb55d354d37a1706a65ab759a7b P 4c5c116a405d4e698eb9b15b3c50a17e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0cb38587aca8444fb40d67e32941ea03 (127.25.211.130:32867): Not found: Tablet not found: e32d0cb55d354d37a1706a65ab759a7b
W20260806 02:55:39.034960 16499 leader_election.cc:341] T e32d0cb55d354d37a1706a65ab759a7b P 4c5c116a405d4e698eb9b15b3c50a17e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7acfb5f0395a47608eac1775aa86e11b (127.25.211.131:39655): Not found: Tablet not found: e32d0cb55d354d37a1706a65ab759a7b
I20260806 02:55:39.035122 16499 leader_election.cc:302] T e32d0cb55d354d37a1706a65ab759a7b P 4c5c116a405d4e698eb9b15b3c50a17e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c5c116a405d4e698eb9b15b3c50a17e; no voters: 0cb38587aca8444fb40d67e32941ea03, 7acfb5f0395a47608eac1775aa86e11b
I20260806 02:55:39.035558 17066 raft_consensus.cc:2726] T e32d0cb55d354d37a1706a65ab759a7b P 4c5c116a405d4e698eb9b15b3c50a17e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 02:55:39.294579 17070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:39.293580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45343 (local address 127.25.211.130:32917)
0806 02:55:39.294095 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:39.294308 (+ 213us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":245,"threads_started":1}
I20260806 02:55:39.757174 17072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:39.756386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45343 (local address 127.25.211.131:54435)
0806 02:55:39.756832 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:39.756947 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":211,"threads_started":1}
I20260806 02:55:39.823684 17074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:39.822926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45343 (local address 127.25.211.129:34363)
0806 02:55:39.823387 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:39.823479 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":231,"threads_started":1}
I20260806 02:55:40.296263 17075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:40.295517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45343 (local address 127.25.211.130:56333)
0806 02:55:40.295998 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:40.296089 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":236,"threads_started":1}
I20260806 02:55:40.597915 17076 raft_consensus.cc:482] T e32d0cb55d354d37a1706a65ab759a7b P 4c5c116a405d4e698eb9b15b3c50a17e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:55:40.598181 17076 raft_consensus.cc:504] T e32d0cb55d354d37a1706a65ab759a7b P 4c5c116a405d4e698eb9b15b3c50a17e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb38587aca8444fb40d67e32941ea03" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 32867 } } peers { permanent_uuid: "4c5c116a405d4e698eb9b15b3c50a17e" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 43767 } } peers { permanent_uuid: "7acfb5f0395a47608eac1775aa86e11b" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 39655 } }
I20260806 02:55:40.598871 17076 leader_election.cc:288] T e32d0cb55d354d37a1706a65ab759a7b P 4c5c116a405d4e698eb9b15b3c50a17e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0cb38587aca8444fb40d67e32941ea03 (127.25.211.130:32867), 7acfb5f0395a47608eac1775aa86e11b (127.25.211.131:39655)
W20260806 02:55:40.599833 16499 leader_election.cc:341] T e32d0cb55d354d37a1706a65ab759a7b P 4c5c116a405d4e698eb9b15b3c50a17e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7acfb5f0395a47608eac1775aa86e11b (127.25.211.131:39655): Not found: Tablet not found: e32d0cb55d354d37a1706a65ab759a7b
W20260806 02:55:40.600009 16498 leader_election.cc:341] T e32d0cb55d354d37a1706a65ab759a7b P 4c5c116a405d4e698eb9b15b3c50a17e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0cb38587aca8444fb40d67e32941ea03 (127.25.211.130:32867): Not found: Tablet not found: e32d0cb55d354d37a1706a65ab759a7b
I20260806 02:55:40.600127 16498 leader_election.cc:302] T e32d0cb55d354d37a1706a65ab759a7b P 4c5c116a405d4e698eb9b15b3c50a17e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c5c116a405d4e698eb9b15b3c50a17e; no voters: 0cb38587aca8444fb40d67e32941ea03, 7acfb5f0395a47608eac1775aa86e11b
I20260806 02:55:40.600353 17076 raft_consensus.cc:2726] T e32d0cb55d354d37a1706a65ab759a7b P 4c5c116a405d4e698eb9b15b3c50a17e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 02:55:40.758812 17079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:40.758017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45343 (local address 127.25.211.131:52377)
0806 02:55:40.758525 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:40.758617 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":275,"threads_started":1}
I20260806 02:55:40.825435 17080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:40.824595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45343 (local address 127.25.211.129:39315)
0806 02:55:40.825136 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:40.825241 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":249,"threads_started":1}
I20260806 02:55:41.297940 17081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:41.297208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45343 (local address 127.25.211.130:49657)
0806 02:55:41.297676 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:41.297773 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":238,"threads_started":1}
I20260806 02:55:41.760540 17083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:41.759803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45343 (local address 127.25.211.131:44851)
0806 02:55:41.760272 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:41.760370 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":241,"threads_started":1}
I20260806 02:55:41.827216 17084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:41.826420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45343 (local address 127.25.211.129:54819)
0806 02:55:41.826931 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:41.827035 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":282,"threads_started":1}
I20260806 02:55:42.299645 17085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:42.298914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45343 (local address 127.25.211.130:56687)
0806 02:55:42.299366 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:42.299457 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":216,"threads_started":1}
I20260806 02:55:42.762467 17086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:42.761592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45343 (local address 127.25.211.131:36473)
0806 02:55:42.762120 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:42.762261 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":261,"threads_started":1}
I20260806 02:55:42.829262 17087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:42.828402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45343 (local address 127.25.211.129:55347)
0806 02:55:42.828958 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:42.829049 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":266,"threads_started":1}
I20260806 02:55:43.301307 17088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:43.300589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45343 (local address 127.25.211.130:36437)
0806 02:55:43.301040 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:43.301129 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":233,"threads_started":1}
I20260806 02:55:43.764247 17089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:43.763455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45343 (local address 127.25.211.131:34493)
0806 02:55:43.763985 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:43.764087 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":301,"threads_started":1}
I20260806 02:55:43.831143 17090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:43.830320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45343 (local address 127.25.211.129:57229)
0806 02:55:43.830845 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:43.830954 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":293,"threads_started":1}
I20260806 02:55:43.925136 17091 raft_consensus.cc:482] T e32d0cb55d354d37a1706a65ab759a7b P 4c5c116a405d4e698eb9b15b3c50a17e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:55:43.925355 17091 raft_consensus.cc:504] T e32d0cb55d354d37a1706a65ab759a7b P 4c5c116a405d4e698eb9b15b3c50a17e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb38587aca8444fb40d67e32941ea03" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 32867 } } peers { permanent_uuid: "4c5c116a405d4e698eb9b15b3c50a17e" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 43767 } } peers { permanent_uuid: "7acfb5f0395a47608eac1775aa86e11b" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 39655 } }
I20260806 02:55:43.926047 17091 leader_election.cc:288] T e32d0cb55d354d37a1706a65ab759a7b P 4c5c116a405d4e698eb9b15b3c50a17e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0cb38587aca8444fb40d67e32941ea03 (127.25.211.130:32867), 7acfb5f0395a47608eac1775aa86e11b (127.25.211.131:39655)
W20260806 02:55:43.927042 16498 leader_election.cc:341] T e32d0cb55d354d37a1706a65ab759a7b P 4c5c116a405d4e698eb9b15b3c50a17e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0cb38587aca8444fb40d67e32941ea03 (127.25.211.130:32867): Not found: Tablet not found: e32d0cb55d354d37a1706a65ab759a7b
W20260806 02:55:43.927213 16499 leader_election.cc:341] T e32d0cb55d354d37a1706a65ab759a7b P 4c5c116a405d4e698eb9b15b3c50a17e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7acfb5f0395a47608eac1775aa86e11b (127.25.211.131:39655): Not found: Tablet not found: e32d0cb55d354d37a1706a65ab759a7b
I20260806 02:55:43.927325 16499 leader_election.cc:302] T e32d0cb55d354d37a1706a65ab759a7b P 4c5c116a405d4e698eb9b15b3c50a17e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c5c116a405d4e698eb9b15b3c50a17e; no voters: 0cb38587aca8444fb40d67e32941ea03, 7acfb5f0395a47608eac1775aa86e11b
I20260806 02:55:43.927546 17091 raft_consensus.cc:2726] T e32d0cb55d354d37a1706a65ab759a7b P 4c5c116a405d4e698eb9b15b3c50a17e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 02:55:44.303298 17092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:44.302351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45343 (local address 127.25.211.130:60667)
0806 02:55:44.303006 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:44.303114 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":389,"threads_started":1}
I20260806 02:55:44.766088 17093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:44.765265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45343 (local address 127.25.211.131:45005)
0806 02:55:44.765797 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:44.765888 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":268,"threads_started":1}
I20260806 02:55:44.833038 17094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:44.832174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45343 (local address 127.25.211.129:38467)
0806 02:55:44.832718 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:44.832819 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":286,"threads_started":1}
I20260806 02:55:45.305275 17095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:45.304357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45343 (local address 127.25.211.130:40845)
0806 02:55:45.304945 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:45.305067 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":288,"threads_started":1}
I20260806 02:55:45.768004 17096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:45.767114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45343 (local address 127.25.211.131:60143)
0806 02:55:45.767689 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:45.767794 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":258,"threads_started":1}
I20260806 02:55:45.834911 17097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:45.834038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45343 (local address 127.25.211.129:36123)
0806 02:55:45.834580 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:45.834727 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":282,"threads_started":1}
I20260806 02:55:46.307224 17098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:46.306405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45343 (local address 127.25.211.130:40277)
0806 02:55:46.306915 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:46.307013 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":245,"threads_started":1}
I20260806 02:55:46.769866 17099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:46.769030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45343 (local address 127.25.211.131:53989)
0806 02:55:46.769568 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:46.769664 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":286,"threads_started":1}
I20260806 02:55:46.778303 17100 raft_consensus.cc:482] T e32d0cb55d354d37a1706a65ab759a7b P 4c5c116a405d4e698eb9b15b3c50a17e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:55:46.778513 17100 raft_consensus.cc:504] T e32d0cb55d354d37a1706a65ab759a7b P 4c5c116a405d4e698eb9b15b3c50a17e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb38587aca8444fb40d67e32941ea03" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 32867 } } peers { permanent_uuid: "4c5c116a405d4e698eb9b15b3c50a17e" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 43767 } } peers { permanent_uuid: "7acfb5f0395a47608eac1775aa86e11b" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 39655 } }
I20260806 02:55:46.779201 17100 leader_election.cc:288] T e32d0cb55d354d37a1706a65ab759a7b P 4c5c116a405d4e698eb9b15b3c50a17e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0cb38587aca8444fb40d67e32941ea03 (127.25.211.130:32867), 7acfb5f0395a47608eac1775aa86e11b (127.25.211.131:39655)
W20260806 02:55:46.780174 16499 leader_election.cc:341] T e32d0cb55d354d37a1706a65ab759a7b P 4c5c116a405d4e698eb9b15b3c50a17e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7acfb5f0395a47608eac1775aa86e11b (127.25.211.131:39655): Not found: Tablet not found: e32d0cb55d354d37a1706a65ab759a7b
W20260806 02:55:46.780400 16498 leader_election.cc:341] T e32d0cb55d354d37a1706a65ab759a7b P 4c5c116a405d4e698eb9b15b3c50a17e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0cb38587aca8444fb40d67e32941ea03 (127.25.211.130:32867): Not found: Tablet not found: e32d0cb55d354d37a1706a65ab759a7b
I20260806 02:55:46.780547 16498 leader_election.cc:302] T e32d0cb55d354d37a1706a65ab759a7b P 4c5c116a405d4e698eb9b15b3c50a17e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c5c116a405d4e698eb9b15b3c50a17e; no voters: 0cb38587aca8444fb40d67e32941ea03, 7acfb5f0395a47608eac1775aa86e11b
I20260806 02:55:46.780813 17100 raft_consensus.cc:2726] T e32d0cb55d354d37a1706a65ab759a7b P 4c5c116a405d4e698eb9b15b3c50a17e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 02:55:46.836773 17102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:46.835831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45343 (local address 127.25.211.129:53725)
0806 02:55:46.836405 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:46.836522 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":299,"threads_started":1}
I20260806 02:55:47.309054 17103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:47.308228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45343 (local address 127.25.211.130:49121)
0806 02:55:47.308783 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:47.308879 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":245,"threads_started":1}
I20260806 02:55:47.771777 17104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:47.770967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45343 (local address 127.25.211.131:46055)
0806 02:55:47.771506 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:47.771608 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":261,"threads_started":1}
I20260806 02:55:47.838672 17105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:47.837821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45343 (local address 127.25.211.129:56931)
0806 02:55:47.838380 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:47.838482 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":330,"threads_started":1}
I20260806 02:55:48.310766 17106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:48.310006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45343 (local address 127.25.211.130:44045)
0806 02:55:48.310480 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:48.310565 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":264,"threads_started":1}
I20260806 02:55:48.773528 17107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:48.772776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45343 (local address 127.25.211.131:42725)
0806 02:55:48.773261 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:48.773346 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":252,"threads_started":1}
I20260806 02:55:48.840698 17108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:48.839741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45343 (local address 127.25.211.129:45999)
0806 02:55:48.840259 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:48.840411 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":286,"threads_started":1}
I20260806 02:55:49.312546 17110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:49.311789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45343 (local address 127.25.211.130:47307)
0806 02:55:49.312273 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:49.312376 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":229,"threads_started":1}
I20260806 02:55:49.775425 17111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:49.774608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45343 (local address 127.25.211.131:58337)
0806 02:55:49.775163 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:49.775255 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":276,"threads_started":1}
I20260806 02:55:49.842574 17112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:49.841741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45343 (local address 127.25.211.129:36651)
0806 02:55:49.842291 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:49.842400 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":275,"threads_started":1}
I20260806 02:55:50.314459 17114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:50.313632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45343 (local address 127.25.211.130:46417)
0806 02:55:50.314183 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:50.314275 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":273,"threads_started":1}
I20260806 02:55:50.777242 17115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:50.776425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45343 (local address 127.25.211.131:37779)
0806 02:55:50.776966 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:50.777065 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":288,"threads_started":1}
I20260806 02:55:50.844328 17117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:50.843537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45343 (local address 127.25.211.129:33973)
0806 02:55:50.844024 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:50.844117 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":250,"threads_started":1}
I20260806 02:55:51.316174 17120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:51.315417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45343 (local address 127.25.211.130:44269)
0806 02:55:51.315896 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:51.315995 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":250,"threads_started":1}
I20260806 02:55:51.778956 17121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:51.778192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45343 (local address 127.25.211.131:37629)
0806 02:55:51.778689 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:51.778786 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":236,"threads_started":1}
I20260806 02:55:51.846427 17123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:51.845411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45343 (local address 127.25.211.129:48213)
0806 02:55:51.845947 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:51.846038 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":292,"threads_started":1}
I20260806 02:55:52.318128 17124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:52.317246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45343 (local address 127.25.211.130:40863)
0806 02:55:52.317802 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:52.317913 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":293,"threads_started":1}
I20260806 02:55:52.780809 17125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:52.779948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45343 (local address 127.25.211.131:40661)
0806 02:55:52.780497 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:52.780597 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":278,"threads_started":1}
I20260806 02:55:52.848654 17126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:52.847846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45343 (local address 127.25.211.129:54491)
0806 02:55:52.848357 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:52.848453 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":251,"threads_started":1}
I20260806 02:55:53.050987 17127 raft_consensus.cc:482] T e32d0cb55d354d37a1706a65ab759a7b P 4c5c116a405d4e698eb9b15b3c50a17e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:55:53.051254 17127 raft_consensus.cc:504] T e32d0cb55d354d37a1706a65ab759a7b P 4c5c116a405d4e698eb9b15b3c50a17e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb38587aca8444fb40d67e32941ea03" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 32867 } } peers { permanent_uuid: "4c5c116a405d4e698eb9b15b3c50a17e" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 43767 } } peers { permanent_uuid: "7acfb5f0395a47608eac1775aa86e11b" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 39655 } }
I20260806 02:55:53.051986 17127 leader_election.cc:288] T e32d0cb55d354d37a1706a65ab759a7b P 4c5c116a405d4e698eb9b15b3c50a17e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0cb38587aca8444fb40d67e32941ea03 (127.25.211.130:32867), 7acfb5f0395a47608eac1775aa86e11b (127.25.211.131:39655)
W20260806 02:55:53.053058 16499 leader_election.cc:341] T e32d0cb55d354d37a1706a65ab759a7b P 4c5c116a405d4e698eb9b15b3c50a17e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7acfb5f0395a47608eac1775aa86e11b (127.25.211.131:39655): Not found: Tablet not found: e32d0cb55d354d37a1706a65ab759a7b
W20260806 02:55:53.053243 16498 leader_election.cc:341] T e32d0cb55d354d37a1706a65ab759a7b P 4c5c116a405d4e698eb9b15b3c50a17e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0cb38587aca8444fb40d67e32941ea03 (127.25.211.130:32867): Not found: Tablet not found: e32d0cb55d354d37a1706a65ab759a7b
I20260806 02:55:53.053376 16498 leader_election.cc:302] T e32d0cb55d354d37a1706a65ab759a7b P 4c5c116a405d4e698eb9b15b3c50a17e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4c5c116a405d4e698eb9b15b3c50a17e; no voters: 0cb38587aca8444fb40d67e32941ea03, 7acfb5f0395a47608eac1775aa86e11b
I20260806 02:55:53.053653 17127 raft_consensus.cc:2726] T e32d0cb55d354d37a1706a65ab759a7b P 4c5c116a405d4e698eb9b15b3c50a17e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 02:55:53.320228 17128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:53.319409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45343 (local address 127.25.211.130:48401)
0806 02:55:53.319927 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:53.320055 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":292,"threads_started":1}
I20260806 02:55:53.762346 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 16409
I20260806 02:55:53.781937 17130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:53.763638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46562 (local address 127.25.211.129:43767)
0806 02:55:53.764593 (+ 955us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:53.764601 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:53.764629 (+ 28us) server_negotiation.cc:407] Connection header received
0806 02:55:53.764694 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:53.764701 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:53.764933 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:53.765061 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:53.766594 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:53.767570 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:53.768880 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:53.769456 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:53.769552 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:53.776052 (+ 6500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:53.776068 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:53.776084 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:53.776134 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:53.779008 (+ 2874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:53.779748 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:53.779755 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:53.779773 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:53.779914 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:53.780292 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:53.780296 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:53.780298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:53.780701 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:53.780974 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:53.781285 (+ 311us) server_negotiation.cc:299] Negotiation successful
0806 02:55:53.781603 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":796,"thread_start_us":675,"threads_started":1}
I20260806 02:55:53.782915 17131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:53.782018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:45343 (local address 127.25.211.131:41091)
0806 02:55:53.782645 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:53.782768 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:45343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":403,"threads_started":1}
I20260806 02:55:53.783454 16663 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:55:53.875298 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 16409
I20260806 02:55:53.908633 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 16677
I20260806 02:55:53.922905 17135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:53.909466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46566 (local address 127.25.211.130:32867)
0806 02:55:53.909952 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:53.909957 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:53.909974 (+ 17us) server_negotiation.cc:407] Connection header received
0806 02:55:53.910031 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:53.910035 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:53.910097 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:53.910250 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:53.911583 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:53.912433 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:53.913760 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:53.914430 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:53.914535 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:53.917364 (+ 2829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:53.917369 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:53.917373 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:53.917424 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:53.920283 (+ 2859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:53.921056 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:53.921061 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:53.921063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:53.921207 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:53.921557 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:53.921560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:53.921562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:53.921853 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:53.922068 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:53.922363 (+ 295us) server_negotiation.cc:299] Negotiation successful
0806 02:55:53.922570 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":267,"threads_started":1}
I20260806 02:55:53.923648 16853 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:55:54.010170 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 16677
I20260806 02:55:54.035284 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 16864
I20260806 02:55:54.049764 17137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:54.036250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50032 (local address 127.25.211.131:39655)
0806 02:55:54.036909 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:54.036914 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:54.036931 (+ 17us) server_negotiation.cc:407] Connection header received
0806 02:55:54.036987 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:54.036990 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:54.037059 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:54.037150 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:54.038829 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:54.039695 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:54.041005 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:54.041465 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:54.041563 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:54.044442 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:54.044447 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:54.044451 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:54.044487 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:54.047297 (+ 2810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:54.048011 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:54.048015 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:54.048017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:54.048148 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:54.048476 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:54.048478 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:54.048480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:54.048774 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:54.049001 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:54.049214 (+ 213us) server_negotiation.cc:299] Negotiation successful
0806 02:55:54.049391 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":343,"threads_started":1}
I20260806 02:55:54.050627 17050 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:55:54.136854 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 16864
2026-08-06T02:55:54Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24758 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:55:54 dist-test-slave-zj8q krb5kdc[17141](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:55:54 dist-test-slave-zj8q krb5kdc[17141](info): set up 2 sockets
Aug 06 02:55:54 dist-test-slave-zj8q krb5kdc[17141](info): commencing operation
krb5kdc: starting...
W20260806 02:55:55.385843 26446 mini_kdc.cc:121] Time spent starting KDC: real 1.202s user 0.000s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Aug 06 02:55:55 dist-test-slave-zj8q krb5kdc[17141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:55:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:55:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.211.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab.
Entry for principal kudu/127.25.211.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab.
I20260806 02:55:55.694121 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:40273
--webserver_interface=127.25.211.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:37079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.211.190:40273
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:55:56.057824 17162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:55:56.058189 17162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:55:56.058328 17162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:55:56.065956 17162 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 02:55:56.066083 17162 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 02:55:56.066140 17162 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 02:55:56.066243 17162 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:55:56.066316 17162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:55:56.066414 17162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:55:56.066475 17162 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:55:56.066550 17162 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:55:56.077273 17162 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:37079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.25.211.190:40273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:40273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.211.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:55:56.078825 17162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:55:56.080300 17162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:55:56.093030 17169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:56.093236 17168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:56.092985 17171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:55:56.094446 17162 server_base.cc:1034] running on GCE node
I20260806 02:55:56.095347 17162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:55:56.096921 17162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:55:56.098210 17162 hybrid_clock.cc:648] HybridClock initialized: now 1785984956098099 us; error 45 us; skew 500 ppm
I20260806 02:55:56.098575 17162 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:55:56 dist-test-slave-zj8q krb5kdc[17141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984956, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:55:56.101950 17162 init.cc:377] Logged in from keytab as kudu/127.25.211.190@KRBTEST.COM (short username kudu)
I20260806 02:55:56.103634 17162 webserver.cc:460] Webserver started at http://127.25.211.190:43535/ using document root <none> and password file <none>
I20260806 02:55:56.104403 17162 fs_manager.cc:359] Metadata directory not provided
I20260806 02:55:56.104517 17162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:55:56.104854 17162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:55:56.108142 17162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/master-0/data/instance:
uuid: "9fbba35f31c947459e6407cf737d30a9"
format_stamp: "Formatted at 2026-08-06 02:55:56 on dist-test-slave-zj8q"
server_key: "56ff814429ca0b8259b1b4798847d1ff"
server_key_iv: "f2e84ea6f9c0cc27182d3c90c9c0601f"
server_key_version: "encryptionkey@0"
I20260806 02:55:56.108996 17162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/master-0/wal/instance:
uuid: "9fbba35f31c947459e6407cf737d30a9"
format_stamp: "Formatted at 2026-08-06 02:55:56 on dist-test-slave-zj8q"
server_key: "56ff814429ca0b8259b1b4798847d1ff"
server_key_iv: "f2e84ea6f9c0cc27182d3c90c9c0601f"
server_key_version: "encryptionkey@0"
I20260806 02:55:56.114734 17162 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260806 02:55:56.118495 17181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:56.119966 17162 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:55:56.120149 17162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/master-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/master-0/wal
uuid: "9fbba35f31c947459e6407cf737d30a9"
format_stamp: "Formatted at 2026-08-06 02:55:56 on dist-test-slave-zj8q"
server_key: "56ff814429ca0b8259b1b4798847d1ff"
server_key_iv: "f2e84ea6f9c0cc27182d3c90c9c0601f"
server_key_version: "encryptionkey@0"
I20260806 02:55:56.120383 17162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:55:56.140733 17162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:55:56.145335 17162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:55:56.145638 17162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:55:56.164954 17162 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.190:40273
I20260806 02:55:56.164927 17258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.190:40273 every 8 connection(s)
I20260806 02:55:56.166476 17162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
I20260806 02:55:56.170353 17259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:55:56.175055 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 17162
I20260806 02:55:56.175252 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/master-0/wal/instance
I20260806 02:55:56.175948 26446 external_mini_cluster.cc:1428] Setting key 7cd5ab6e03e021a8739b9e53a26dfbd5
I20260806 02:55:56.181653 17259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9: Bootstrap starting.
I20260806 02:55:56.184947 17259 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9: Neither blocks nor log segments found. Creating new log.
I20260806 02:55:56.185999 17259 log.cc:824] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9: Log is configured to *not* fsync() on all Append() calls
I20260806 02:55:56.189046 17259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9: No bootstrap required, opened a new log
Aug 06 02:55:56 dist-test-slave-zj8q krb5kdc[17141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:55:56.197317 17259 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fbba35f31c947459e6407cf737d30a9" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 40273 } }
I20260806 02:55:56.197716 17259 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:55:56.197840 17259 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fbba35f31c947459e6407cf737d30a9, State: Initialized, Role: FOLLOWER
I20260806 02:55:56.198516 17259 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fbba35f31c947459e6407cf737d30a9" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 40273 } }
I20260806 02:55:56.198782 17259 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:55:56.198925 17259 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:55:56.199090 17259 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:55:56.199368 17262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:56.178124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42572 (local address 127.25.211.190:40273)
0806 02:55:56.178972 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:56.178988 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:56.179034 (+ 46us) server_negotiation.cc:407] Connection header received
0806 02:55:56.180792 (+ 1758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:56.180818 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:56.181221 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:56.181590 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:56.183303 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:56.184653 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:56.186133 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:56.186938 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:56.187130 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:56.191694 (+ 4564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:56.191703 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:56.191723 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:56.191771 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:56.194984 (+ 3213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:56.195730 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:56.195741 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:56.195748 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:56.195942 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:56.196327 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:56.196334 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:56.196336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:56.196912 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:56.197182 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:56.197690 (+ 508us) server_negotiation.cc:299] Negotiation successful
0806 02:55:56.198184 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":342,"threads_started":1}
I20260806 02:55:56.202122 17259 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fbba35f31c947459e6407cf737d30a9" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 40273 } }
I20260806 02:55:56.202841 17259 leader_election.cc:302] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9fbba35f31c947459e6407cf737d30a9; no voters:
I20260806 02:55:56.203517 17259 leader_election.cc:288] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:55:56.203630 17264 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:55:56.204694 17264 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9 [term 1 LEADER]: Becoming Leader. State: Replica: 9fbba35f31c947459e6407cf737d30a9, State: Running, Role: LEADER
I20260806 02:55:56.205454 17264 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fbba35f31c947459e6407cf737d30a9" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 40273 } }
I20260806 02:55:56.205785 17259 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:55:56.208559 17265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9fbba35f31c947459e6407cf737d30a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fbba35f31c947459e6407cf737d30a9" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 40273 } } }
I20260806 02:55:56.208989 17265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9 [sys.catalog]: This master's current role is: LEADER
I20260806 02:55:56.208552 17267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fbba35f31c947459e6407cf737d30a9. Latest consensus state: current_term: 1 leader_uuid: "9fbba35f31c947459e6407cf737d30a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fbba35f31c947459e6407cf737d30a9" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 40273 } } }
I20260806 02:55:56.209133 17267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9 [sys.catalog]: This master's current role is: LEADER
I20260806 02:55:56.211344 17274 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:55:56.216872 17274 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:55:56.227072 17274 catalog_manager.cc:1347] Generated new cluster ID: 918343e9b6c54837a2d7994365cdaa9d
I20260806 02:55:56.227212 17274 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:55:56.238199 17274 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:55:56.239807 17274 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:55:56.247572 17274 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9: Generated new TSK 0
I20260806 02:55:56.248427 17274 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.211.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab.
Entry for principal kudu/127.25.211.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab.
I20260806 02:55:56.320281 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.129:0
--local_ip_for_outbound_sockets=127.25.211.129
--webserver_interface=127.25.211.129
--webserver_port=0
--tserver_master_addrs=127.25.211.190:40273
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab
--principal=kudu/127.25.211.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:37079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:55:56.673312 17290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:55:56.673661 17290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:55:56.673802 17290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:55:56.681421 17290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:55:56.681567 17290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:55:56.681733 17290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.129
I20260806 02:55:56.691895 17290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:37079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab
--principal=kudu/127.25.211.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.211.129
--webserver_port=0
--tserver_master_addrs=127.25.211.190:40273
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.129
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:55:56.693281 17290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:55:56.694629 17290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:55:56.707932 17297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:56.708148 17302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:56.707912 17299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:55:56.709162 17290 server_base.cc:1034] running on GCE node
I20260806 02:55:56.709815 17290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:55:56.710803 17290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:55:56.712003 17290 hybrid_clock.cc:648] HybridClock initialized: now 1785984956711944 us; error 54 us; skew 500 ppm
I20260806 02:55:56.712323 17290 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:55:56 dist-test-slave-zj8q krb5kdc[17141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984956, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:55:56.715607 17290 init.cc:377] Logged in from keytab as kudu/127.25.211.129@KRBTEST.COM (short username kudu)
I20260806 02:55:56.717063 17290 webserver.cc:460] Webserver started at http://127.25.211.129:33697/ using document root <none> and password file <none>
I20260806 02:55:56.717746 17290 fs_manager.cc:359] Metadata directory not provided
I20260806 02:55:56.717854 17290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:55:56.718135 17290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:55:56.721239 17290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-0/data/instance:
uuid: "e510bcdf18a54a4392a3392d53e84d4b"
format_stamp: "Formatted at 2026-08-06 02:55:56 on dist-test-slave-zj8q"
server_key: "94f068353ac3a82a0be9510dacbc6285"
server_key_iv: "528daddd51df4da6c848dbf8e452a33d"
server_key_version: "encryptionkey@0"
I20260806 02:55:56.721999 17290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-0/wal/instance:
uuid: "e510bcdf18a54a4392a3392d53e84d4b"
format_stamp: "Formatted at 2026-08-06 02:55:56 on dist-test-slave-zj8q"
server_key: "94f068353ac3a82a0be9510dacbc6285"
server_key_iv: "528daddd51df4da6c848dbf8e452a33d"
server_key_version: "encryptionkey@0"
I20260806 02:55:56.727713 17290 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260806 02:55:56.731374 17312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:56.732738 17290 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 02:55:56.732908 17290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-0/wal
uuid: "e510bcdf18a54a4392a3392d53e84d4b"
format_stamp: "Formatted at 2026-08-06 02:55:56 on dist-test-slave-zj8q"
server_key: "94f068353ac3a82a0be9510dacbc6285"
server_key_iv: "528daddd51df4da6c848dbf8e452a33d"
server_key_version: "encryptionkey@0"
I20260806 02:55:56.733114 17290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:55:56.760886 17290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:55:56.765642 17290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:55:56.765939 17290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:55:56.766985 17290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:55:56.768421 17290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:55:56.768498 17290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:56.768620 17290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:55:56.768674 17290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:56.800331 17290 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.129:41493
I20260806 02:55:56.800380 17496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.129:41493 every 8 connection(s)
I20260806 02:55:56.801630 17290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
I20260806 02:55:56.802690 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 17290
I20260806 02:55:56.802948 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-0/wal/instance
I20260806 02:55:56.803840 26446 external_mini_cluster.cc:1428] Setting key beda421f10e9820021c37b27869648af
Aug 06 02:55:56 dist-test-slave-zj8q krb5kdc[17141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984956, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.129@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:55:56.824028 17500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:56.804611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:53575 (local address 127.25.211.190:40273)
0806 02:55:56.805093 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:56.805099 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:56.806294 (+ 1195us) server_negotiation.cc:407] Connection header received
0806 02:55:56.808239 (+ 1945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:56.808244 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:56.808337 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:56.808452 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:56.809955 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:56.811058 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:56.812767 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:56.813349 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:56.813474 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:56.817361 (+ 3887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:56.817367 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:56.817371 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:56.817408 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:56.820229 (+ 2821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:56.821080 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:56.821086 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:56.821090 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:56.821264 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:56.821913 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:56.821918 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:56.821922 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:56.822377 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:56.822529 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:56.823411 (+ 882us) server_negotiation.cc:299] Negotiation successful
0806 02:55:56.823665 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":248,"threads_started":1}
I20260806 02:55:56.825011 17501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:56.805244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:40273 (local address 127.25.211.129:53575)
0806 02:55:56.806097 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:56.806173 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0806 02:55:56.807965 (+ 1792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:55:56.808647 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:55:56.808671 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:55:56.809189 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:55:56.809781 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:56.809799 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:56.811257 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:56.811261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:56.812627 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:56.812645 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:56.813633 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:56.813640 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:56.813851 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:56.814543 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:55:56.814572 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:55:56.817068 (+ 2496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:55:56.820413 (+ 3345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:56.820422 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:56.820438 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:56.820938 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:56.821432 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:56.821436 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:56.821439 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:56.821777 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:56.822534 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:55:56.822541 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:55:56.822943 (+ 402us) client_negotiation.cc:769] Sending connection context
0806 02:55:56.823222 (+ 279us) client_negotiation.cc:240] Negotiation successful
0806 02:55:56.823680 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":363,"threads_started":1}
I20260806 02:55:56.827208 17497 heartbeater.cc:347] Connected to a master server at 127.25.211.190:40273
I20260806 02:55:56.827612 17497 heartbeater.cc:464] Registering TS with master...
I20260806 02:55:56.828413 17497 heartbeater.cc:511] Master 127.25.211.190:40273 requested a full tablet report, sending...
I20260806 02:55:56.830812 17210 ts_manager.cc:194] Registered new tserver with Master: e510bcdf18a54a4392a3392d53e84d4b (127.25.211.129:41493)
I20260806 02:55:56.832419 17210 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.211.129@KRBTEST.COM'} at 127.25.211.129:53575
WARNING: no policy specified for kudu/127.25.211.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab.
Entry for principal kudu/127.25.211.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab.
I20260806 02:55:56.873261 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.130:0
--local_ip_for_outbound_sockets=127.25.211.130
--webserver_interface=127.25.211.130
--webserver_port=0
--tserver_master_addrs=127.25.211.190:40273
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab
--principal=kudu/127.25.211.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:37079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:55:57.217809 17505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:55:57.218115 17505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:55:57.218264 17505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:55:57.225996 17505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:55:57.226135 17505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:55:57.226328 17505 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.130
I20260806 02:55:57.236738 17505 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:37079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab
--principal=kudu/127.25.211.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.211.130
--webserver_port=0
--tserver_master_addrs=127.25.211.190:40273
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.130
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:55:57.238402 17505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:55:57.239912 17505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:55:57.253569 17512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:57.254063 17513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:57.254798 17516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:55:57.255484 17505 server_base.cc:1034] running on GCE node
I20260806 02:55:57.256150 17505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:55:57.257160 17505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:55:57.258370 17505 hybrid_clock.cc:648] HybridClock initialized: now 1785984957258314 us; error 51 us; skew 500 ppm
I20260806 02:55:57.258695 17505 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:55:57 dist-test-slave-zj8q krb5kdc[17141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984957, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:55:57.261951 17505 init.cc:377] Logged in from keytab as kudu/127.25.211.130@KRBTEST.COM (short username kudu)
I20260806 02:55:57.263695 17505 webserver.cc:460] Webserver started at http://127.25.211.130:34423/ using document root <none> and password file <none>
I20260806 02:55:57.264454 17505 fs_manager.cc:359] Metadata directory not provided
I20260806 02:55:57.264577 17505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:55:57.264889 17505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:55:57.268010 17505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-1/data/instance:
uuid: "ad3f7123cd8d4dcebd11197c8bfd5d39"
format_stamp: "Formatted at 2026-08-06 02:55:57 on dist-test-slave-zj8q"
server_key: "a5d418862eaf3a46b7a9bac6cc8d1112"
server_key_iv: "165d0e598acfe0545342fadf5333ab47"
server_key_version: "encryptionkey@0"
I20260806 02:55:57.268939 17505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-1/wal/instance:
uuid: "ad3f7123cd8d4dcebd11197c8bfd5d39"
format_stamp: "Formatted at 2026-08-06 02:55:57 on dist-test-slave-zj8q"
server_key: "a5d418862eaf3a46b7a9bac6cc8d1112"
server_key_iv: "165d0e598acfe0545342fadf5333ab47"
server_key_version: "encryptionkey@0"
I20260806 02:55:57.274742 17505 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260806 02:55:57.278496 17525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:57.279867 17505 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:55:57.280028 17505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-1/wal
uuid: "ad3f7123cd8d4dcebd11197c8bfd5d39"
format_stamp: "Formatted at 2026-08-06 02:55:57 on dist-test-slave-zj8q"
server_key: "a5d418862eaf3a46b7a9bac6cc8d1112"
server_key_iv: "165d0e598acfe0545342fadf5333ab47"
server_key_version: "encryptionkey@0"
I20260806 02:55:57.280227 17505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:55:57.306329 17505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:55:57.311019 17505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:55:57.311335 17505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:55:57.312402 17505 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:55:57.313885 17505 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:55:57.313969 17505 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:57.314116 17505 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:55:57.314196 17505 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260806 02:55:57.344676 17505 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.130:42087
I20260806 02:55:57.344717 17681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.130:42087 every 8 connection(s)
I20260806 02:55:57.346006 17505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
I20260806 02:55:57.354574 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 17505
I20260806 02:55:57.354763 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-1/wal/instance
I20260806 02:55:57.355213 26446 external_mini_cluster.cc:1428] Setting key 8ffe32ac0485106c9d8390ece6a73b38
Aug 06 02:55:57 dist-test-slave-zj8q krb5kdc[17141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984957, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.130@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:55:57.366858 17685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:57.348611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:40923 (local address 127.25.211.190:40273)
0806 02:55:57.349092 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:57.349100 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:57.350054 (+ 954us) server_negotiation.cc:407] Connection header received
0806 02:55:57.351878 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:57.351883 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:57.351964 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:57.352059 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:57.353488 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:57.354264 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:57.355998 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:57.356580 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:57.356712 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:57.360371 (+ 3659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:57.360378 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:57.360382 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:57.360422 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:57.363527 (+ 3105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:57.364316 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:57.364322 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:57.364326 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:57.364498 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:57.364933 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:57.364938 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:57.364941 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:57.365335 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:57.365481 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:57.366260 (+ 779us) server_negotiation.cc:299] Negotiation successful
0806 02:55:57.366495 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":224,"threads_started":1}
I20260806 02:55:57.367803 17686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:57.349152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:40273 (local address 127.25.211.130:40923)
0806 02:55:57.349892 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:57.349940 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 02:55:57.351613 (+ 1673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:55:57.352243 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:55:57.352251 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:55:57.352750 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:55:57.353349 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:57.353368 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:57.354425 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:57.354429 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:57.355838 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:57.355853 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:57.356832 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:57.356838 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:57.357132 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:57.357787 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:55:57.357813 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:55:57.360107 (+ 2294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:55:57.363686 (+ 3579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:57.363695 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:57.363710 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:57.364185 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:57.364598 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:57.364601 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:57.364604 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:57.364830 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:57.365457 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:55:57.365463 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:55:57.365855 (+ 392us) client_negotiation.cc:769] Sending connection context
0806 02:55:57.366107 (+ 252us) client_negotiation.cc:240] Negotiation successful
0806 02:55:57.366593 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":274,"threads_started":1}
I20260806 02:55:57.369619 17683 heartbeater.cc:347] Connected to a master server at 127.25.211.190:40273
I20260806 02:55:57.369977 17683 heartbeater.cc:464] Registering TS with master...
I20260806 02:55:57.370805 17683 heartbeater.cc:511] Master 127.25.211.190:40273 requested a full tablet report, sending...
I20260806 02:55:57.372339 17210 ts_manager.cc:194] Registered new tserver with Master: ad3f7123cd8d4dcebd11197c8bfd5d39 (127.25.211.130:42087)
I20260806 02:55:57.373188 17210 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.211.130@KRBTEST.COM'} at 127.25.211.130:40923
WARNING: no policy specified for kudu/127.25.211.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab.
Entry for principal kudu/127.25.211.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab.
I20260806 02:55:57.420825 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.131:0
--local_ip_for_outbound_sockets=127.25.211.131
--webserver_interface=127.25.211.131
--webserver_port=0
--tserver_master_addrs=127.25.211.190:40273
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab
--principal=kudu/127.25.211.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:37079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:55:57.773563 17689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:55:57.773887 17689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:55:57.773989 17689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:55:57.781493 17689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:55:57.781616 17689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:55:57.781733 17689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.131
I20260806 02:55:57.792038 17689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:37079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab
--principal=kudu/127.25.211.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.211.131
--webserver_port=0
--tserver_master_addrs=127.25.211.190:40273
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.131
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:55:57.793361 17689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:55:57.794749 17689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:55:57.808458 17694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:57.808416 17698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:55:57.808494 17695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:55:57.809087 17689 server_base.cc:1034] running on GCE node
I20260806 02:55:57.809681 17689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:55:57.810647 17689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:55:57.811847 17689 hybrid_clock.cc:648] HybridClock initialized: now 1785984957811797 us; error 41 us; skew 500 ppm
I20260806 02:55:57.812188 17689 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:55:57 dist-test-slave-zj8q krb5kdc[17141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984957, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:55:57.815338 17689 init.cc:377] Logged in from keytab as kudu/127.25.211.131@KRBTEST.COM (short username kudu)
I20260806 02:55:57.816836 17689 webserver.cc:460] Webserver started at http://127.25.211.131:40627/ using document root <none> and password file <none>
I20260806 02:55:57.817548 17689 fs_manager.cc:359] Metadata directory not provided
I20260806 02:55:57.817672 17689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:55:57.817986 17689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:55:57.820986 17689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-2/data/instance:
uuid: "6b00a6aadf6240b0b34a3c2594e2e0d7"
format_stamp: "Formatted at 2026-08-06 02:55:57 on dist-test-slave-zj8q"
server_key: "a856e286e26af2352781b931775f42e6"
server_key_iv: "772930b073f49b4447b74462bb61c8d3"
server_key_version: "encryptionkey@0"
I20260806 02:55:57.821733 17689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-2/wal/instance:
uuid: "6b00a6aadf6240b0b34a3c2594e2e0d7"
format_stamp: "Formatted at 2026-08-06 02:55:57 on dist-test-slave-zj8q"
server_key: "a856e286e26af2352781b931775f42e6"
server_key_iv: "772930b073f49b4447b74462bb61c8d3"
server_key_version: "encryptionkey@0"
I20260806 02:55:57.827250 17689 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260806 02:55:57.831147 17706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:57.832532 17689 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260806 02:55:57.832707 17689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-2/wal
uuid: "6b00a6aadf6240b0b34a3c2594e2e0d7"
format_stamp: "Formatted at 2026-08-06 02:55:57 on dist-test-slave-zj8q"
server_key: "a856e286e26af2352781b931775f42e6"
server_key_iv: "772930b073f49b4447b74462bb61c8d3"
server_key_version: "encryptionkey@0"
I20260806 02:55:57.832919 17689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:55:57.835686 17497 heartbeater.cc:503] Master 127.25.211.190:40273 was elected leader, sending a full tablet report...
I20260806 02:55:57.883235 17689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:55:57.887974 17689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:55:57.888293 17689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:55:57.889329 17689 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:55:57.890889 17689 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:55:57.890972 17689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:57.891109 17689 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:55:57.891165 17689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:55:57.923671 17689 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.131:43105
I20260806 02:55:57.923753 17868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.131:43105 every 8 connection(s)
I20260806 02:55:57.925053 17689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
I20260806 02:55:57.932832 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 17689
I20260806 02:55:57.933059 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-2/wal/instance
I20260806 02:55:57.933657 26446 external_mini_cluster.cc:1428] Setting key 827cc8acc840d81f0dab931b5d7568cc
Aug 06 02:55:57 dist-test-slave-zj8q krb5kdc[17141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984957, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.131@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:55:57.946705 17871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:57.927819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:56179 (local address 127.25.211.190:40273)
0806 02:55:57.928302 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:57.928307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:57.929364 (+ 1057us) server_negotiation.cc:407] Connection header received
0806 02:55:57.931401 (+ 2037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:57.931407 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:57.931505 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:57.931605 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:57.933626 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:57.934665 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:57.936713 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:57.937254 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:57.937349 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:57.940990 (+ 3641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:57.940996 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:57.940999 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:57.941038 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:57.943631 (+ 2593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:57.944363 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:57.944367 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:57.944369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:57.944533 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:57.944908 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:57.944911 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:57.944913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:57.945193 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:57.945314 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:57.946196 (+ 882us) server_negotiation.cc:299] Negotiation successful
0806 02:55:57.946405 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":256,"threads_started":1}
I20260806 02:55:57.947714 17872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:57.928407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:40273 (local address 127.25.211.131:56179)
0806 02:55:57.929169 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:57.929218 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 02:55:57.931084 (+ 1866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:55:57.931890 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:55:57.931899 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:55:57.932531 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:55:57.933411 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:57.933440 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:57.934847 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:57.934851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:57.936547 (+ 1696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:57.936565 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:57.937459 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:57.937464 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:57.937734 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:57.938502 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:55:57.938534 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:55:57.940775 (+ 2241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:55:57.943795 (+ 3020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:57.943803 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:57.943816 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:57.944271 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:57.944616 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:55:57.944619 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:55:57.944622 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:55:57.944826 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:55:57.945289 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:55:57.945294 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:55:57.945669 (+ 375us) client_negotiation.cc:769] Sending connection context
0806 02:55:57.945935 (+ 266us) client_negotiation.cc:240] Negotiation successful
0806 02:55:57.946444 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":289,"threads_started":1}
I20260806 02:55:57.949229 17869 heartbeater.cc:347] Connected to a master server at 127.25.211.190:40273
I20260806 02:55:57.949543 17869 heartbeater.cc:464] Registering TS with master...
I20260806 02:55:57.950363 17869 heartbeater.cc:511] Master 127.25.211.190:40273 requested a full tablet report, sending...
I20260806 02:55:57.951953 17210 ts_manager.cc:194] Registered new tserver with Master: 6b00a6aadf6240b0b34a3c2594e2e0d7 (127.25.211.131:43105)
I20260806 02:55:57.952801 17210 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.211.131@KRBTEST.COM'} at 127.25.211.131:56179
I20260806 02:55:57.961982 26446 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:55:57.981050 17871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:57.966869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42574 (local address 127.25.211.190:40273)
0806 02:55:57.967223 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:57.967230 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:57.967890 (+ 660us) server_negotiation.cc:407] Connection header received
0806 02:55:57.968090 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:57.968094 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:57.968169 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:57.968274 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:57.969936 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:57.970790 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:57.972325 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:57.972822 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:57.972938 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:57.975753 (+ 2815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:57.975759 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:57.975762 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:57.975805 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:57.978535 (+ 2730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:57.979248 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:57.979253 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:57.979255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:57.979389 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:57.979702 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:57.979706 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:57.979708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:57.980049 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:57.980252 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:57.980557 (+ 305us) server_negotiation.cc:299] Negotiation successful
0806 02:55:57.980751 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260806 02:55:57.985441 17210 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42574:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:55:57.992609 17210 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:55:58.021093 17883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:58.006096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.131:43105 (local address 127.0.0.1:35030)
0806 02:55:58.007597 (+ 1501us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:58.007658 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0806 02:55:58.007875 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:55:58.009832 (+ 1957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:55:58.009840 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:55:58.009899 (+ 59us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:55:58.010536 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:58.010556 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:58.012894 (+ 2338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:58.012901 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:58.015583 (+ 2682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:58.015605 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:58.019908 (+ 4303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:58.019915 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:58.020159 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:58.020229 (+ 70us) client_negotiation.cc:769] Sending connection context
0806 02:55:58.020363 (+ 134us) client_negotiation.cc:240] Negotiation successful
0806 02:55:58.020584 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1182,"mutex_wait_us":39,"spinlock_wait_cycles":14464,"thread_start_us":2008,"threads_started":1}
I20260806 02:55:58.022346 17884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:58.006346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35030 (local address 127.25.211.131:43105)
0806 02:55:58.008848 (+ 2502us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:58.008857 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:58.008896 (+ 39us) server_negotiation.cc:407] Connection header received
0806 02:55:58.009004 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:58.009011 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:58.009299 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:58.009510 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:55:58.010794 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:58.012713 (+ 1919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:58.015812 (+ 3099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:58.019339 (+ 3527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:58.019544 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:58.019756 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:58.021559 (+ 1803us) server_negotiation.cc:299] Negotiation successful
0806 02:55:58.021773 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2322,"thread_start_us":2149,"threads_started":1}
I20260806 02:55:58.022809 17882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:58.006234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:41493 (local address 127.0.0.1:49160)
0806 02:55:58.008427 (+ 2193us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:58.008446 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:55:58.008569 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:55:58.009837 (+ 1268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:55:58.009841 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:55:58.009890 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:55:58.010669 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:58.010689 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:58.017415 (+ 6726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:58.017421 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:58.019486 (+ 2065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:58.019500 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:58.021830 (+ 2330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:58.021838 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:58.022019 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:58.022054 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 02:55:58.022140 (+ 86us) client_negotiation.cc:240] Negotiation successful
0806 02:55:58.022371 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2075,"mutex_wait_us":84,"spinlock_wait_cycles":896,"thread_start_us":969,"threads_started":1}
I20260806 02:55:58.023896 17885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:58.006996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49160 (local address 127.25.211.129:41493)
0806 02:55:58.009264 (+ 2268us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:58.009271 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:58.009296 (+ 25us) server_negotiation.cc:407] Connection header received
0806 02:55:58.009398 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:58.009406 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:58.009648 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:58.009869 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:55:58.010919 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:58.017247 (+ 6328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:58.019727 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:58.021241 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:58.021447 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:58.021643 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:58.023354 (+ 1711us) server_negotiation.cc:299] Negotiation successful
0806 02:55:58.023551 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2097,"thread_start_us":1947,"threads_started":1}
I20260806 02:55:58.028054 17887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:58.010204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.130:42087 (local address 127.0.0.1:45406)
0806 02:55:58.016949 (+ 6745us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:58.016972 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 02:55:58.017121 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:55:58.019106 (+ 1985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:55:58.019112 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:55:58.019142 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:55:58.019575 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:58.019586 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:58.023362 (+ 3776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:58.023367 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:58.025409 (+ 2042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:58.025428 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:58.027267 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:58.027274 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:58.027441 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:58.027471 (+ 30us) client_negotiation.cc:769] Sending connection context
0806 02:55:58.027554 (+ 83us) client_negotiation.cc:240] Negotiation successful
0806 02:55:58.027694 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6605,"thread_start_us":7001,"threads_started":1}
I20260806 02:55:58.027562 17391 tablet_service.cc:1467] Processing CreateTablet for tablet 9f57bb9606524fc88996b9094320f2e4 (DEFAULT_TABLE table=test-table [id=4a791e31c7f54fc9878f5f5af31a258f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:55:58.027405 17778 tablet_service.cc:1467] Processing CreateTablet for tablet 9f57bb9606524fc88996b9094320f2e4 (DEFAULT_TABLE table=test-table [id=4a791e31c7f54fc9878f5f5af31a258f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:55:58.028123 17888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:58.010483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45406 (local address 127.25.211.130:42087)
0806 02:55:58.018403 (+ 7920us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:58.018413 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:58.018447 (+ 34us) server_negotiation.cc:407] Connection header received
0806 02:55:58.018556 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:58.018565 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:58.018854 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:58.019774 (+ 920us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:55:58.020042 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:58.023177 (+ 3135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:58.025566 (+ 2389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:58.027034 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:58.027203 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:58.027382 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:58.027620 (+ 238us) server_negotiation.cc:299] Negotiation successful
0806 02:55:58.027813 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7722,"thread_start_us":7566,"threads_started":1}
I20260806 02:55:58.029127 17778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f57bb9606524fc88996b9094320f2e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:55:58.029146 17391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f57bb9606524fc88996b9094320f2e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:55:58.031497 17591 tablet_service.cc:1467] Processing CreateTablet for tablet 9f57bb9606524fc88996b9094320f2e4 (DEFAULT_TABLE table=test-table [id=4a791e31c7f54fc9878f5f5af31a258f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:55:58.032831 17591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f57bb9606524fc88996b9094320f2e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:55:58.047636 17896 tablet_bootstrap.cc:492] T 9f57bb9606524fc88996b9094320f2e4 P 6b00a6aadf6240b0b34a3c2594e2e0d7: Bootstrap starting.
I20260806 02:55:58.048288 17893 tablet_bootstrap.cc:492] T 9f57bb9606524fc88996b9094320f2e4 P ad3f7123cd8d4dcebd11197c8bfd5d39: Bootstrap starting.
I20260806 02:55:58.048755 17895 tablet_bootstrap.cc:492] T 9f57bb9606524fc88996b9094320f2e4 P e510bcdf18a54a4392a3392d53e84d4b: Bootstrap starting.
I20260806 02:55:58.051510 17893 tablet_bootstrap.cc:654] T 9f57bb9606524fc88996b9094320f2e4 P ad3f7123cd8d4dcebd11197c8bfd5d39: Neither blocks nor log segments found. Creating new log.
I20260806 02:55:58.051734 17896 tablet_bootstrap.cc:654] T 9f57bb9606524fc88996b9094320f2e4 P 6b00a6aadf6240b0b34a3c2594e2e0d7: Neither blocks nor log segments found. Creating new log.
I20260806 02:55:58.052452 17895 tablet_bootstrap.cc:654] T 9f57bb9606524fc88996b9094320f2e4 P e510bcdf18a54a4392a3392d53e84d4b: Neither blocks nor log segments found. Creating new log.
I20260806 02:55:58.052664 17893 log.cc:824] T 9f57bb9606524fc88996b9094320f2e4 P ad3f7123cd8d4dcebd11197c8bfd5d39: Log is configured to *not* fsync() on all Append() calls
I20260806 02:55:58.053052 17896 log.cc:824] T 9f57bb9606524fc88996b9094320f2e4 P 6b00a6aadf6240b0b34a3c2594e2e0d7: Log is configured to *not* fsync() on all Append() calls
I20260806 02:55:58.053743 17895 log.cc:824] T 9f57bb9606524fc88996b9094320f2e4 P e510bcdf18a54a4392a3392d53e84d4b: Log is configured to *not* fsync() on all Append() calls
I20260806 02:55:58.056097 17893 tablet_bootstrap.cc:492] T 9f57bb9606524fc88996b9094320f2e4 P ad3f7123cd8d4dcebd11197c8bfd5d39: No bootstrap required, opened a new log
I20260806 02:55:58.056420 17893 ts_tablet_manager.cc:1397] T 9f57bb9606524fc88996b9094320f2e4 P ad3f7123cd8d4dcebd11197c8bfd5d39: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260806 02:55:58.057173 17895 tablet_bootstrap.cc:492] T 9f57bb9606524fc88996b9094320f2e4 P e510bcdf18a54a4392a3392d53e84d4b: No bootstrap required, opened a new log
I20260806 02:55:58.057197 17896 tablet_bootstrap.cc:492] T 9f57bb9606524fc88996b9094320f2e4 P 6b00a6aadf6240b0b34a3c2594e2e0d7: No bootstrap required, opened a new log
I20260806 02:55:58.057456 17895 ts_tablet_manager.cc:1397] T 9f57bb9606524fc88996b9094320f2e4 P e510bcdf18a54a4392a3392d53e84d4b: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.003s
I20260806 02:55:58.057473 17896 ts_tablet_manager.cc:1397] T 9f57bb9606524fc88996b9094320f2e4 P 6b00a6aadf6240b0b34a3c2594e2e0d7: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260806 02:55:58.065014 17895 raft_consensus.cc:348] T 9f57bb9606524fc88996b9094320f2e4 P e510bcdf18a54a4392a3392d53e84d4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b00a6aadf6240b0b34a3c2594e2e0d7" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 43105 } } peers { permanent_uuid: "e510bcdf18a54a4392a3392d53e84d4b" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 41493 } } peers { permanent_uuid: "ad3f7123cd8d4dcebd11197c8bfd5d39" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 42087 } }
I20260806 02:55:58.065469 17895 raft_consensus.cc:374] T 9f57bb9606524fc88996b9094320f2e4 P e510bcdf18a54a4392a3392d53e84d4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:55:58.065631 17895 raft_consensus.cc:729] T 9f57bb9606524fc88996b9094320f2e4 P e510bcdf18a54a4392a3392d53e84d4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e510bcdf18a54a4392a3392d53e84d4b, State: Initialized, Role: FOLLOWER
I20260806 02:55:58.066346 17895 consensus_queue.cc:260] T 9f57bb9606524fc88996b9094320f2e4 P e510bcdf18a54a4392a3392d53e84d4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b00a6aadf6240b0b34a3c2594e2e0d7" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 43105 } } peers { permanent_uuid: "e510bcdf18a54a4392a3392d53e84d4b" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 41493 } } peers { permanent_uuid: "ad3f7123cd8d4dcebd11197c8bfd5d39" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 42087 } }
I20260806 02:55:58.068332 17895 ts_tablet_manager.cc:1428] T 9f57bb9606524fc88996b9094320f2e4 P e510bcdf18a54a4392a3392d53e84d4b: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260806 02:55:58.068032 17896 raft_consensus.cc:348] T 9f57bb9606524fc88996b9094320f2e4 P 6b00a6aadf6240b0b34a3c2594e2e0d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b00a6aadf6240b0b34a3c2594e2e0d7" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 43105 } } peers { permanent_uuid: "e510bcdf18a54a4392a3392d53e84d4b" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 41493 } } peers { permanent_uuid: "ad3f7123cd8d4dcebd11197c8bfd5d39" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 42087 } }
I20260806 02:55:58.068495 17896 raft_consensus.cc:374] T 9f57bb9606524fc88996b9094320f2e4 P 6b00a6aadf6240b0b34a3c2594e2e0d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:55:58.068681 17896 raft_consensus.cc:729] T 9f57bb9606524fc88996b9094320f2e4 P 6b00a6aadf6240b0b34a3c2594e2e0d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b00a6aadf6240b0b34a3c2594e2e0d7, State: Initialized, Role: FOLLOWER
I20260806 02:55:58.067649 17893 raft_consensus.cc:348] T 9f57bb9606524fc88996b9094320f2e4 P ad3f7123cd8d4dcebd11197c8bfd5d39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b00a6aadf6240b0b34a3c2594e2e0d7" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 43105 } } peers { permanent_uuid: "e510bcdf18a54a4392a3392d53e84d4b" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 41493 } } peers { permanent_uuid: "ad3f7123cd8d4dcebd11197c8bfd5d39" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 42087 } }
I20260806 02:55:58.069028 17893 raft_consensus.cc:374] T 9f57bb9606524fc88996b9094320f2e4 P ad3f7123cd8d4dcebd11197c8bfd5d39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:55:58.069173 17893 raft_consensus.cc:729] T 9f57bb9606524fc88996b9094320f2e4 P ad3f7123cd8d4dcebd11197c8bfd5d39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad3f7123cd8d4dcebd11197c8bfd5d39, State: Initialized, Role: FOLLOWER
I20260806 02:55:58.069504 17896 consensus_queue.cc:260] T 9f57bb9606524fc88996b9094320f2e4 P 6b00a6aadf6240b0b34a3c2594e2e0d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b00a6aadf6240b0b34a3c2594e2e0d7" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 43105 } } peers { permanent_uuid: "e510bcdf18a54a4392a3392d53e84d4b" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 41493 } } peers { permanent_uuid: "ad3f7123cd8d4dcebd11197c8bfd5d39" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 42087 } }
I20260806 02:55:58.069794 17893 consensus_queue.cc:260] T 9f57bb9606524fc88996b9094320f2e4 P ad3f7123cd8d4dcebd11197c8bfd5d39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b00a6aadf6240b0b34a3c2594e2e0d7" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 43105 } } peers { permanent_uuid: "e510bcdf18a54a4392a3392d53e84d4b" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 41493 } } peers { permanent_uuid: "ad3f7123cd8d4dcebd11197c8bfd5d39" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 42087 } }
I20260806 02:55:58.070814 17869 heartbeater.cc:503] Master 127.25.211.190:40273 was elected leader, sending a full tablet report...
I20260806 02:55:58.071512 17896 ts_tablet_manager.cc:1428] T 9f57bb9606524fc88996b9094320f2e4 P 6b00a6aadf6240b0b34a3c2594e2e0d7: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260806 02:55:58.073784 17683 heartbeater.cc:503] Master 127.25.211.190:40273 was elected leader, sending a full tablet report...
I20260806 02:55:58.075366 17893 ts_tablet_manager.cc:1428] T 9f57bb9606524fc88996b9094320f2e4 P ad3f7123cd8d4dcebd11197c8bfd5d39: Time spent starting tablet: real 0.019s user 0.014s sys 0.004s
W20260806 02:55:58.098124 17684 tablet.cc:2369] T 9f57bb9606524fc88996b9094320f2e4 P ad3f7123cd8d4dcebd11197c8bfd5d39: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:55:58.166900 17907 raft_consensus.cc:482] T 9f57bb9606524fc88996b9094320f2e4 P 6b00a6aadf6240b0b34a3c2594e2e0d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:55:58.167289 17907 raft_consensus.cc:504] T 9f57bb9606524fc88996b9094320f2e4 P 6b00a6aadf6240b0b34a3c2594e2e0d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b00a6aadf6240b0b34a3c2594e2e0d7" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 43105 } } peers { permanent_uuid: "e510bcdf18a54a4392a3392d53e84d4b" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 41493 } } peers { permanent_uuid: "ad3f7123cd8d4dcebd11197c8bfd5d39" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 42087 } }
I20260806 02:55:58.169458 17907 leader_election.cc:288] T 9f57bb9606524fc88996b9094320f2e4 P 6b00a6aadf6240b0b34a3c2594e2e0d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e510bcdf18a54a4392a3392d53e84d4b (127.25.211.129:41493), ad3f7123cd8d4dcebd11197c8bfd5d39 (127.25.211.130:42087)
I20260806 02:55:58.176074 17885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:58.170061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:41157 (local address 127.25.211.129:41493)
0806 02:55:58.170304 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:58.170311 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:58.170332 (+ 21us) server_negotiation.cc:407] Connection header received
0806 02:55:58.170406 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:58.170411 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:58.170484 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:58.170590 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:55:58.171219 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:58.172313 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:58.174278 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:58.175097 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:58.175260 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:58.175347 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:58.175669 (+ 322us) server_negotiation.cc:299] Negotiation successful
0806 02:55:58.175825 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260806 02:55:58.176069 17872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:58.169738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:41493 (local address 127.25.211.131:41157)
0806 02:55:58.170072 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:58.170093 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 02:55:58.170263 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:55:58.170809 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:55:58.170814 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:55:58.170846 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:55:58.171053 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:58.171066 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:58.172485 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:58.172490 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:58.174083 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:58.174097 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:58.175263 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:58.175269 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:58.175469 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:58.175504 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 02:55:58.175607 (+ 103us) client_negotiation.cc:240] Negotiation successful
0806 02:55:58.175759 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":25}
W20260806 02:55:58.176790 17870 tablet.cc:2369] T 9f57bb9606524fc88996b9094320f2e4 P 6b00a6aadf6240b0b34a3c2594e2e0d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:55:58.177590 17427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f57bb9606524fc88996b9094320f2e4" candidate_uuid: "6b00a6aadf6240b0b34a3c2594e2e0d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e510bcdf18a54a4392a3392d53e84d4b" is_pre_election: true
I20260806 02:55:58.177912 17888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:58.170841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:57625 (local address 127.25.211.130:42087)
0806 02:55:58.171035 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:58.171042 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:58.171061 (+ 19us) server_negotiation.cc:407] Connection header received
0806 02:55:58.171141 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:58.171145 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:58.171232 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:58.171345 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:55:58.172004 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:58.173032 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:58.175557 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:58.176741 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:58.176898 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:58.176974 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:58.177543 (+ 569us) server_negotiation.cc:299] Negotiation successful
0806 02:55:58.177706 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 02:55:58.178256 17427 raft_consensus.cc:2454] T 9f57bb9606524fc88996b9094320f2e4 P e510bcdf18a54a4392a3392d53e84d4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b00a6aadf6240b0b34a3c2594e2e0d7 in term 0.
I20260806 02:55:58.178594 17913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:58.169880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.130:42087 (local address 127.25.211.131:57625)
0806 02:55:58.170835 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0806 02:55:58.170851 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:55:58.170955 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:55:58.171368 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:55:58.171372 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:55:58.171399 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:55:58.171795 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:58.171810 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:58.173192 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:58.173198 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:58.175375 (+ 2177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:55:58.175391 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:55:58.177141 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:55:58.177148 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:55:58.177334 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:58.177369 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 02:55:58.178082 (+ 713us) client_negotiation.cc:240] Negotiation successful
0806 02:55:58.178259 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":839,"mutex_wait_us":3,"thread_start_us":646,"threads_started":1}
I20260806 02:55:58.179093 17710 leader_election.cc:302] T 9f57bb9606524fc88996b9094320f2e4 P 6b00a6aadf6240b0b34a3c2594e2e0d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b00a6aadf6240b0b34a3c2594e2e0d7, e510bcdf18a54a4392a3392d53e84d4b; no voters:
I20260806 02:55:58.179507 17907 raft_consensus.cc:2781] T 9f57bb9606524fc88996b9094320f2e4 P 6b00a6aadf6240b0b34a3c2594e2e0d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:55:58.179675 17907 raft_consensus.cc:482] T 9f57bb9606524fc88996b9094320f2e4 P 6b00a6aadf6240b0b34a3c2594e2e0d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:55:58.179772 17907 raft_consensus.cc:3037] T 9f57bb9606524fc88996b9094320f2e4 P 6b00a6aadf6240b0b34a3c2594e2e0d7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:55:58.179832 17621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f57bb9606524fc88996b9094320f2e4" candidate_uuid: "6b00a6aadf6240b0b34a3c2594e2e0d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad3f7123cd8d4dcebd11197c8bfd5d39" is_pre_election: true
I20260806 02:55:58.180354 17621 raft_consensus.cc:2454] T 9f57bb9606524fc88996b9094320f2e4 P ad3f7123cd8d4dcebd11197c8bfd5d39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b00a6aadf6240b0b34a3c2594e2e0d7 in term 0.
I20260806 02:55:58.183100 17907 raft_consensus.cc:504] T 9f57bb9606524fc88996b9094320f2e4 P 6b00a6aadf6240b0b34a3c2594e2e0d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b00a6aadf6240b0b34a3c2594e2e0d7" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 43105 } } peers { permanent_uuid: "e510bcdf18a54a4392a3392d53e84d4b" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 41493 } } peers { permanent_uuid: "ad3f7123cd8d4dcebd11197c8bfd5d39" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 42087 } }
I20260806 02:55:58.183830 17907 leader_election.cc:288] T 9f57bb9606524fc88996b9094320f2e4 P 6b00a6aadf6240b0b34a3c2594e2e0d7 [CANDIDATE]: Term 1 election: Requested vote from peers e510bcdf18a54a4392a3392d53e84d4b (127.25.211.129:41493), ad3f7123cd8d4dcebd11197c8bfd5d39 (127.25.211.130:42087)
I20260806 02:55:58.184379 17427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f57bb9606524fc88996b9094320f2e4" candidate_uuid: "6b00a6aadf6240b0b34a3c2594e2e0d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e510bcdf18a54a4392a3392d53e84d4b"
I20260806 02:55:58.184374 17621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f57bb9606524fc88996b9094320f2e4" candidate_uuid: "6b00a6aadf6240b0b34a3c2594e2e0d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad3f7123cd8d4dcebd11197c8bfd5d39"
I20260806 02:55:58.184588 17427 raft_consensus.cc:3037] T 9f57bb9606524fc88996b9094320f2e4 P e510bcdf18a54a4392a3392d53e84d4b [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:55:58.184602 17621 raft_consensus.cc:3037] T 9f57bb9606524fc88996b9094320f2e4 P ad3f7123cd8d4dcebd11197c8bfd5d39 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:55:58.187032 17427 raft_consensus.cc:2454] T 9f57bb9606524fc88996b9094320f2e4 P e510bcdf18a54a4392a3392d53e84d4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b00a6aadf6240b0b34a3c2594e2e0d7 in term 1.
I20260806 02:55:58.187052 17621 raft_consensus.cc:2454] T 9f57bb9606524fc88996b9094320f2e4 P ad3f7123cd8d4dcebd11197c8bfd5d39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b00a6aadf6240b0b34a3c2594e2e0d7 in term 1.
I20260806 02:55:58.187508 17712 leader_election.cc:302] T 9f57bb9606524fc88996b9094320f2e4 P 6b00a6aadf6240b0b34a3c2594e2e0d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b00a6aadf6240b0b34a3c2594e2e0d7, ad3f7123cd8d4dcebd11197c8bfd5d39; no voters:
I20260806 02:55:58.187868 17907 raft_consensus.cc:2781] T 9f57bb9606524fc88996b9094320f2e4 P 6b00a6aadf6240b0b34a3c2594e2e0d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:55:58.188326 17907 raft_consensus.cc:686] T 9f57bb9606524fc88996b9094320f2e4 P 6b00a6aadf6240b0b34a3c2594e2e0d7 [term 1 LEADER]: Becoming Leader. State: Replica: 6b00a6aadf6240b0b34a3c2594e2e0d7, State: Running, Role: LEADER
I20260806 02:55:58.188809 17907 consensus_queue.cc:237] T 9f57bb9606524fc88996b9094320f2e4 P 6b00a6aadf6240b0b34a3c2594e2e0d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b00a6aadf6240b0b34a3c2594e2e0d7" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 43105 } } peers { permanent_uuid: "e510bcdf18a54a4392a3392d53e84d4b" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 41493 } } peers { permanent_uuid: "ad3f7123cd8d4dcebd11197c8bfd5d39" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 42087 } }
I20260806 02:55:58.194072 17210 catalog_manager.cc:5495] T 9f57bb9606524fc88996b9094320f2e4 P 6b00a6aadf6240b0b34a3c2594e2e0d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b00a6aadf6240b0b34a3c2594e2e0d7 (127.25.211.131). New cstate: current_term: 1 leader_uuid: "6b00a6aadf6240b0b34a3c2594e2e0d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b00a6aadf6240b0b34a3c2594e2e0d7" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 43105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e510bcdf18a54a4392a3392d53e84d4b" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 41493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad3f7123cd8d4dcebd11197c8bfd5d39" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 42087 } health_report { overall_health: UNKNOWN } } }
I20260806 02:55:58.222875 17884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:58.217100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35042 (local address 127.25.211.131:43105)
0806 02:55:58.217328 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:58.217335 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:58.217416 (+ 81us) server_negotiation.cc:407] Connection header received
0806 02:55:58.217644 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:58.217648 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:58.217737 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:58.217875 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:55:58.218333 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:58.219054 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:58.220657 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:58.221074 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:58.221194 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:58.221709 (+ 515us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:55:58.222303 (+ 594us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:55:58.222472 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:58.222523 (+ 51us) server_negotiation.cc:299] Negotiation successful
0806 02:55:58.222612 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260806 02:55:58.231410 17621 raft_consensus.cc:1261] T 9f57bb9606524fc88996b9094320f2e4 P ad3f7123cd8d4dcebd11197c8bfd5d39 [term 1 FOLLOWER]: Refusing update from remote peer 6b00a6aadf6240b0b34a3c2594e2e0d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:55:58.231410 17427 raft_consensus.cc:1261] T 9f57bb9606524fc88996b9094320f2e4 P e510bcdf18a54a4392a3392d53e84d4b [term 1 FOLLOWER]: Refusing update from remote peer 6b00a6aadf6240b0b34a3c2594e2e0d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:55:58.232419 17915 consensus_queue.cc:1035] T 9f57bb9606524fc88996b9094320f2e4 P 6b00a6aadf6240b0b34a3c2594e2e0d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad3f7123cd8d4dcebd11197c8bfd5d39" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 42087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:55:58.232774 17907 consensus_queue.cc:1035] T 9f57bb9606524fc88996b9094320f2e4 P 6b00a6aadf6240b0b34a3c2594e2e0d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e510bcdf18a54a4392a3392d53e84d4b" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 41493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:55:58.246907 17918 mvcc.cc:205] Tried to move back new op lower bound from 7315394388905492480 to 7315394388745441280. Current Snapshot: MvccSnapshot[applied={T|T < 7315394388905492480}]
I20260806 02:55:58.247241 17919 mvcc.cc:205] Tried to move back new op lower bound from 7315394388905492480 to 7315394388745441280. Current Snapshot: MvccSnapshot[applied={T|T < 7315394388905492480}]
I20260806 02:55:58.262387 17209 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:55:58.262691 17209 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:55:58.266994 17209 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9: Sending DeleteTablet for 3 replicas of tablet 9f57bb9606524fc88996b9094320f2e4
I20260806 02:55:58.268296 17591 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f57bb9606524fc88996b9094320f2e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:55:58 UTC) from {username='kudu', principal='kudu/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:45406
I20260806 02:55:58.268318 17391 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f57bb9606524fc88996b9094320f2e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:55:58 UTC) from {username='kudu', principal='kudu/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:49160
I20260806 02:55:58.268316 17778 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f57bb9606524fc88996b9094320f2e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:55:58 UTC) from {username='kudu', principal='kudu/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:35030
I20260806 02:55:58.272022 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 17290
I20260806 02:55:58.279767 17932 tablet_replica.cc:330] T 9f57bb9606524fc88996b9094320f2e4 P ad3f7123cd8d4dcebd11197c8bfd5d39: stopping tablet replica
I20260806 02:55:58.280570 17932 raft_consensus.cc:2229] T 9f57bb9606524fc88996b9094320f2e4 P ad3f7123cd8d4dcebd11197c8bfd5d39 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:55:58.281320 17932 raft_consensus.cc:2258] T 9f57bb9606524fc88996b9094320f2e4 P ad3f7123cd8d4dcebd11197c8bfd5d39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:55:58.284126 17932 ts_tablet_manager.cc:1905] T 9f57bb9606524fc88996b9094320f2e4 P ad3f7123cd8d4dcebd11197c8bfd5d39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:55:58.285723 17936 tablet_replica.cc:330] T 9f57bb9606524fc88996b9094320f2e4 P e510bcdf18a54a4392a3392d53e84d4b: stopping tablet replica
I20260806 02:55:58.286111 17936 raft_consensus.cc:2229] T 9f57bb9606524fc88996b9094320f2e4 P e510bcdf18a54a4392a3392d53e84d4b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:55:58.286605 17936 raft_consensus.cc:2258] T 9f57bb9606524fc88996b9094320f2e4 P e510bcdf18a54a4392a3392d53e84d4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:55:58.287392 17934 tablet_replica.cc:330] T 9f57bb9606524fc88996b9094320f2e4 P 6b00a6aadf6240b0b34a3c2594e2e0d7: stopping tablet replica
I20260806 02:55:58.287740 17934 raft_consensus.cc:2229] T 9f57bb9606524fc88996b9094320f2e4 P 6b00a6aadf6240b0b34a3c2594e2e0d7 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:55:58.288338 17934 raft_consensus.cc:2258] T 9f57bb9606524fc88996b9094320f2e4 P 6b00a6aadf6240b0b34a3c2594e2e0d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:55:58.292520 17936 ts_tablet_manager.cc:1905] T 9f57bb9606524fc88996b9094320f2e4 P e510bcdf18a54a4392a3392d53e84d4b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:55:58.294799 17932 ts_tablet_manager.cc:1918] T 9f57bb9606524fc88996b9094320f2e4 P ad3f7123cd8d4dcebd11197c8bfd5d39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:55:58.296473 17934 ts_tablet_manager.cc:1905] T 9f57bb9606524fc88996b9094320f2e4 P 6b00a6aadf6240b0b34a3c2594e2e0d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:55:58.299775 17936 ts_tablet_manager.cc:1918] T 9f57bb9606524fc88996b9094320f2e4 P e510bcdf18a54a4392a3392d53e84d4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:55:58.300010 17936 log.cc:1196] T 9f57bb9606524fc88996b9094320f2e4 P e510bcdf18a54a4392a3392d53e84d4b: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-0/wal/wals/9f57bb9606524fc88996b9094320f2e4
I20260806 02:55:58.300148 17932 log.cc:1196] T 9f57bb9606524fc88996b9094320f2e4 P ad3f7123cd8d4dcebd11197c8bfd5d39: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-1/wal/wals/9f57bb9606524fc88996b9094320f2e4
I20260806 02:55:58.300624 17936 ts_tablet_manager.cc:1939] T 9f57bb9606524fc88996b9094320f2e4 P e510bcdf18a54a4392a3392d53e84d4b: Deleting consensus metadata
I20260806 02:55:58.300853 17932 ts_tablet_manager.cc:1939] T 9f57bb9606524fc88996b9094320f2e4 P ad3f7123cd8d4dcebd11197c8bfd5d39: Deleting consensus metadata
I20260806 02:55:58.302853 17186 catalog_manager.cc:4841] TS e510bcdf18a54a4392a3392d53e84d4b (127.25.211.129:41493): tablet 9f57bb9606524fc88996b9094320f2e4 (table test-table [id=4a791e31c7f54fc9878f5f5af31a258f]) successfully deleted
Aug 06 02:55:58 dist-test-slave-zj8q krb5kdc[17141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.129@KRBTEST.COM
I20260806 02:55:58.304915 17188 catalog_manager.cc:4841] TS ad3f7123cd8d4dcebd11197c8bfd5d39 (127.25.211.130:42087): tablet 9f57bb9606524fc88996b9094320f2e4 (table test-table [id=4a791e31c7f54fc9878f5f5af31a258f]) successfully deleted
I20260806 02:55:58.307056 17934 ts_tablet_manager.cc:1918] T 9f57bb9606524fc88996b9094320f2e4 P 6b00a6aadf6240b0b34a3c2594e2e0d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:55:58.307333 17934 log.cc:1196] T 9f57bb9606524fc88996b9094320f2e4 P 6b00a6aadf6240b0b34a3c2594e2e0d7: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-2/wal/wals/9f57bb9606524fc88996b9094320f2e4
I20260806 02:55:58.307927 17934 ts_tablet_manager.cc:1939] T 9f57bb9606524fc88996b9094320f2e4 P 6b00a6aadf6240b0b34a3c2594e2e0d7: Deleting consensus metadata
I20260806 02:55:58.309752 17183 catalog_manager.cc:4841] TS 6b00a6aadf6240b0b34a3c2594e2e0d7 (127.25.211.131:43105): tablet 9f57bb9606524fc88996b9094320f2e4 (table test-table [id=4a791e31c7f54fc9878f5f5af31a258f]) successfully deleted
I20260806 02:55:58.312111 17885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:58.273638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49176 (local address 127.25.211.129:41493)
0806 02:55:58.273848 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:58.273856 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:58.291127 (+ 17271us) server_negotiation.cc:407] Connection header received
0806 02:55:58.291725 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:58.291731 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:58.291822 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:58.291910 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:58.294200 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:58.295442 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:58.297643 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:58.298406 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:58.298509 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:58.304514 (+ 6005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:58.304529 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:58.304548 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:58.304593 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:58.308605 (+ 4012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:58.309778 (+ 1173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:58.309789 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:58.309793 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:58.309972 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:58.310496 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:58.310499 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:58.310501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:58.310892 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:58.311164 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:58.311597 (+ 433us) server_negotiation.cc:299] Negotiation successful
0806 02:55:58.311814 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260806 02:55:58.313088 17487 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:55:58.422282 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 17290
I20260806 02:55:58.451860 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 17505
Aug 06 02:55:58 dist-test-slave-zj8q krb5kdc[17141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.130@KRBTEST.COM
I20260806 02:55:58.469362 17888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:58.452895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45420 (local address 127.25.211.130:42087)
0806 02:55:58.453213 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:58.453220 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:58.453243 (+ 23us) server_negotiation.cc:407] Connection header received
0806 02:55:58.453394 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:58.453398 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:58.453464 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:58.453567 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:58.455008 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:58.455783 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:58.457189 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:58.457607 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:58.457729 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:58.462924 (+ 5195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:58.462944 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:58.462964 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:58.463018 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:58.466244 (+ 3226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:58.467224 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:58.467232 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:58.467234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:58.467365 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:58.467763 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:58.467766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:58.467768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:58.468206 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:58.468438 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:58.468829 (+ 391us) server_negotiation.cc:299] Negotiation successful
0806 02:55:58.469018 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260806 02:55:58.470049 17676 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:55:58.565253 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 17505
I20260806 02:55:58.597798 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 17689
Aug 06 02:55:58 dist-test-slave-zj8q krb5kdc[17141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.131@KRBTEST.COM
I20260806 02:55:58.617522 17884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:55:58.598706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35052 (local address 127.25.211.131:43105)
0806 02:55:58.598865 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0806 02:55:58.598870 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:55:58.598889 (+ 19us) server_negotiation.cc:407] Connection header received
0806 02:55:58.599018 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:55:58.599022 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:55:58.599113 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:55:58.599209 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:55:58.601073 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:58.601922 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:58.603959 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:55:58.604533 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:55:58.604645 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:55:58.611054 (+ 6409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:55:58.611074 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:55:58.611097 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:55:58.611166 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:55:58.614425 (+ 3259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:58.615442 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:58.615450 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:58.615452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:58.615630 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:55:58.616076 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:55:58.616083 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:55:58.616085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:55:58.616449 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:55:58.616719 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0806 02:55:58.617057 (+ 338us) server_negotiation.cc:299] Negotiation successful
0806 02:55:58.617236 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 02:55:58.618480 17864 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:55:58.744338 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 17689
I20260806 02:55:58.770058 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 17162
I20260806 02:55:58.771327 17254 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:55:58.884351 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 17162
I20260806 02:56:18.910534 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:40273
--webserver_interface=127.25.211.190
--webserver_port=43535
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:37079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.211.190:40273
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:56:19.272688 17944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:56:19.273005 17944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:56:19.273121 17944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:56:19.280628 17944 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 02:56:19.280751 17944 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 02:56:19.280799 17944 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 02:56:19.280880 17944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:56:19.280956 17944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:56:19.281054 17944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:56:19.281113 17944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:56:19.281167 17944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:56:19.291440 17944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:37079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.25.211.190:40273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:40273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.211.190
--webserver_port=43535
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:56:19.292829 17944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:56:19.294299 17944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:56:19.320807 17953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:56:19.320811 17957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:56:19.321269 17944 server_base.cc:1034] running on GCE node
W20260806 02:56:19.321801 17952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:56:19.322942 17944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:56:19.324687 17944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:56:19.325944 17944 hybrid_clock.cc:648] HybridClock initialized: now 1785984979325870 us; error 42 us; skew 500 ppm
I20260806 02:56:19.326419 17944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:56:19 dist-test-slave-zj8q krb5kdc[17141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984979, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:56:19.330581 17944 init.cc:377] Logged in from keytab as kudu/127.25.211.190@KRBTEST.COM (short username kudu)
I20260806 02:56:19.332489 17944 webserver.cc:460] Webserver started at http://127.25.211.190:43535/ using document root <none> and password file <none>
I20260806 02:56:19.333464 17944 fs_manager.cc:359] Metadata directory not provided
I20260806 02:56:19.333604 17944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:56:19.340866 17944 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.001s
I20260806 02:56:19.344413 17966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:56:19.346031 17944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:56:19.346247 17944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/master-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/master-0/wal
uuid: "9fbba35f31c947459e6407cf737d30a9"
format_stamp: "Formatted at 2026-08-06 02:55:56 on dist-test-slave-zj8q"
server_key: "56ff814429ca0b8259b1b4798847d1ff"
server_key_iv: "f2e84ea6f9c0cc27182d3c90c9c0601f"
server_key_version: "encryptionkey@0"
I20260806 02:56:19.347407 17944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:56:19.367319 17944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:56:19.372030 17944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:56:19.372349 17944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:56:19.391474 17944 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.190:40273
I20260806 02:56:19.391510 18037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.190:40273 every 8 connection(s)
I20260806 02:56:19.393069 17944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
I20260806 02:56:19.395743 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 17944
I20260806 02:56:19.396690 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.129:41493
--local_ip_for_outbound_sockets=127.25.211.129
--tserver_master_addrs=127.25.211.190:40273
--webserver_port=33697
--webserver_interface=127.25.211.129
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab
--principal=kudu/127.25.211.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:37079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:56:19.400557 18038 sys_catalog.cc:263] Verifying existing consensus state
I20260806 02:56:19.404369 18038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9: Bootstrap starting.
I20260806 02:56:19.425100 18038 log.cc:824] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9: Log is configured to *not* fsync() on all Append() calls
I20260806 02:56:19.438411 18038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:56:19.439010 18038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9: Bootstrap complete.
I20260806 02:56:19.448266 18038 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fbba35f31c947459e6407cf737d30a9" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 40273 } }
I20260806 02:56:19.449112 18038 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fbba35f31c947459e6407cf737d30a9, State: Initialized, Role: FOLLOWER
I20260806 02:56:19.449753 18038 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fbba35f31c947459e6407cf737d30a9" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 40273 } }
I20260806 02:56:19.449985 18038 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:56:19.450106 18038 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:56:19.450320 18038 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9 [term 1 FOLLOWER]: Advancing to term 2
I20260806 02:56:19.454308 18038 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fbba35f31c947459e6407cf737d30a9" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 40273 } }
I20260806 02:56:19.454838 18038 leader_election.cc:302] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9fbba35f31c947459e6407cf737d30a9; no voters:
I20260806 02:56:19.455466 18038 leader_election.cc:288] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 02:56:19.455602 18047 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 02:56:19.456748 18047 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9 [term 2 LEADER]: Becoming Leader. State: Replica: 9fbba35f31c947459e6407cf737d30a9, State: Running, Role: LEADER
I20260806 02:56:19.457409 18047 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fbba35f31c947459e6407cf737d30a9" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 40273 } }
I20260806 02:56:19.457791 18038 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:56:19.460400 18049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9fbba35f31c947459e6407cf737d30a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fbba35f31c947459e6407cf737d30a9" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 40273 } } }
I20260806 02:56:19.461143 18049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9 [sys.catalog]: This master's current role is: LEADER
I20260806 02:56:19.460642 18050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fbba35f31c947459e6407cf737d30a9. Latest consensus state: current_term: 2 leader_uuid: "9fbba35f31c947459e6407cf737d30a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fbba35f31c947459e6407cf737d30a9" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 40273 } } }
I20260806 02:56:19.461716 18050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9 [sys.catalog]: This master's current role is: LEADER
I20260806 02:56:19.473609 18056 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:56:19.483057 18056 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:56:19.484735 18056 catalog_manager.cc:1259] Loaded cluster ID: 918343e9b6c54837a2d7994365cdaa9d
I20260806 02:56:19.484923 18056 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:56:19.489410 18056 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:56:19.489739 18072 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 02:56:19.491084 18056 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9: Loaded TSK: 0
I20260806 02:56:19.510955 18056 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9: Generated new TSK 1
I20260806 02:56:19.511364 18056 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 02:56:19.784029 18042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:56:19.784335 18042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:56:19.784472 18042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:56:19.791810 18042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:56:19.791950 18042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:56:19.792101 18042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.129
I20260806 02:56:19.802145 18042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:37079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab
--principal=kudu/127.25.211.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.129:41493
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.211.129
--webserver_port=33697
--tserver_master_addrs=127.25.211.190:40273
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.129
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:56:19.803562 18042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:56:19.804953 18042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:56:19.821033 18086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:56:19.821012 18081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:56:19.821475 18083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:56:19.822016 18042 server_base.cc:1034] running on GCE node
I20260806 02:56:19.822818 18042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:56:19.824021 18042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:56:19.825273 18042 hybrid_clock.cc:648] HybridClock initialized: now 1785984979825200 us; error 59 us; skew 500 ppm
I20260806 02:56:19.825686 18042 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:56:19 dist-test-slave-zj8q krb5kdc[17141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984979, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:56:19.829185 18042 init.cc:377] Logged in from keytab as kudu/127.25.211.129@KRBTEST.COM (short username kudu)
I20260806 02:56:19.830974 18042 webserver.cc:460] Webserver started at http://127.25.211.129:33697/ using document root <none> and password file <none>
I20260806 02:56:19.831704 18042 fs_manager.cc:359] Metadata directory not provided
I20260806 02:56:19.831842 18042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:56:19.837679 18042 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:56:19.840643 18095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:56:19.842031 18042 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260806 02:56:19.842244 18042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-0/wal
uuid: "e510bcdf18a54a4392a3392d53e84d4b"
format_stamp: "Formatted at 2026-08-06 02:55:56 on dist-test-slave-zj8q"
server_key: "94f068353ac3a82a0be9510dacbc6285"
server_key_iv: "528daddd51df4da6c848dbf8e452a33d"
server_key_version: "encryptionkey@0"
I20260806 02:56:19.843343 18042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:56:19.862581 18042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:56:19.867213 18042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:56:19.867492 18042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:56:19.868568 18042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:56:19.869968 18042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:56:19.870069 18042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:56:19.870219 18042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:56:19.870297 18042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:56:19.901516 18042 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.129:41493
I20260806 02:56:19.901589 18272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.129:41493 every 8 connection(s)
I20260806 02:56:19.903036 18042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
I20260806 02:56:19.908811 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 18042
I20260806 02:56:19.909618 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.130:42087
--local_ip_for_outbound_sockets=127.25.211.130
--tserver_master_addrs=127.25.211.190:40273
--webserver_port=34423
--webserver_interface=127.25.211.130
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab
--principal=kudu/127.25.211.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:37079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:56:19 dist-test-slave-zj8q krb5kdc[17141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984979, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.129@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:56:19.926461 18275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:19.905922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:59757 (local address 127.25.211.190:40273)
0806 02:56:19.906520 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0806 02:56:19.906531 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:56:19.907062 (+ 531us) server_negotiation.cc:407] Connection header received
0806 02:56:19.909268 (+ 2206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:56:19.909287 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:56:19.909722 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:56:19.910113 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:56:19.911490 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:56:19.913257 (+ 1767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:56:19.914860 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:56:19.915535 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:56:19.915711 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:56:19.919300 (+ 3589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:56:19.919308 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:56:19.919322 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:56:19.919358 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:56:19.922046 (+ 2688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:56:19.922772 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:56:19.922779 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:56:19.922784 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:56:19.922948 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:56:19.923373 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:56:19.923376 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:56:19.923378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:56:19.923853 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:56:19.924033 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0806 02:56:19.924803 (+ 770us) server_negotiation.cc:299] Negotiation successful
0806 02:56:19.925183 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":275,"threads_started":1}
I20260806 02:56:19.926461 18276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:19.906066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:40273 (local address 127.25.211.129:59757)
0806 02:56:19.906884 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:19.906936 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 02:56:19.908660 (+ 1724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:56:19.910260 (+ 1600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:56:19.910267 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:56:19.910781 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:56:19.911296 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:56:19.911313 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:56:19.913420 (+ 2107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:56:19.913424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:56:19.914696 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:56:19.914731 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:56:19.915777 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:56:19.915785 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:56:19.916042 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:56:19.916714 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:56:19.916737 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:56:19.919025 (+ 2288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:56:19.922208 (+ 3183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:56:19.922218 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:56:19.922232 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:56:19.922632 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:56:19.923055 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:56:19.923060 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:56:19.923064 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:56:19.923262 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:56:19.923985 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:56:19.923993 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:56:19.924319 (+ 326us) client_negotiation.cc:769] Sending connection context
0806 02:56:19.924552 (+ 233us) client_negotiation.cc:240] Negotiation successful
0806 02:56:19.925103 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":314,"threads_started":1}
I20260806 02:56:19.929517 18273 heartbeater.cc:347] Connected to a master server at 127.25.211.190:40273
I20260806 02:56:19.929803 18273 heartbeater.cc:464] Registering TS with master...
I20260806 02:56:19.930538 18273 heartbeater.cc:511] Master 127.25.211.190:40273 requested a full tablet report, sending...
I20260806 02:56:19.933015 17993 ts_manager.cc:194] Registered new tserver with Master: e510bcdf18a54a4392a3392d53e84d4b (127.25.211.129:41493)
I20260806 02:56:19.935168 17993 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.211.129@KRBTEST.COM'} at 127.25.211.129:59757
W20260806 02:56:20.262336 18277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:56:20.262640 18277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:56:20.262754 18277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:56:20.270243 18277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:56:20.270368 18277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:56:20.270491 18277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.130
I20260806 02:56:20.281136 18277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:37079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab
--principal=kudu/127.25.211.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.130:42087
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.211.130
--webserver_port=34423
--tserver_master_addrs=127.25.211.190:40273
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.130
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:56:20.282519 18277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:56:20.283919 18277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:56:20.297672 18288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:56:20.297678 18285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:56:20.297701 18284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:56:20.298578 18277 server_base.cc:1034] running on GCE node
I20260806 02:56:20.299235 18277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:56:20.300212 18277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:56:20.301422 18277 hybrid_clock.cc:648] HybridClock initialized: now 1785984980301368 us; error 41 us; skew 500 ppm
I20260806 02:56:20.301756 18277 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:56:20 dist-test-slave-zj8q krb5kdc[17141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984980, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:56:20.305061 18277 init.cc:377] Logged in from keytab as kudu/127.25.211.130@KRBTEST.COM (short username kudu)
I20260806 02:56:20.306663 18277 webserver.cc:460] Webserver started at http://127.25.211.130:34423/ using document root <none> and password file <none>
I20260806 02:56:20.307366 18277 fs_manager.cc:359] Metadata directory not provided
I20260806 02:56:20.307487 18277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:56:20.313092 18277 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:56:20.315945 18306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:56:20.317409 18277 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 02:56:20.317570 18277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-1/wal
uuid: "ad3f7123cd8d4dcebd11197c8bfd5d39"
format_stamp: "Formatted at 2026-08-06 02:55:57 on dist-test-slave-zj8q"
server_key: "a5d418862eaf3a46b7a9bac6cc8d1112"
server_key_iv: "165d0e598acfe0545342fadf5333ab47"
server_key_version: "encryptionkey@0"
I20260806 02:56:20.318619 18277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:56:20.333693 18277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:56:20.338073 18277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:56:20.338404 18277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:56:20.339428 18277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:56:20.340920 18277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:56:20.341022 18277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:56:20.341132 18277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:56:20.341210 18277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:56:20.373427 18277 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.130:42087
I20260806 02:56:20.373490 18470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.130:42087 every 8 connection(s)
I20260806 02:56:20.375092 18277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
I20260806 02:56:20.381809 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 18277
I20260806 02:56:20.382489 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.131:43105
--local_ip_for_outbound_sockets=127.25.211.131
--tserver_master_addrs=127.25.211.190:40273
--webserver_port=40627
--webserver_interface=127.25.211.131
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab
--principal=kudu/127.25.211.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:37079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 02:56:20 dist-test-slave-zj8q krb5kdc[17141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984980, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.130@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:56:20.397204 18275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:20.378016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:56015 (local address 127.25.211.190:40273)
0806 02:56:20.378227 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0806 02:56:20.378231 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:56:20.379455 (+ 1224us) server_negotiation.cc:407] Connection header received
0806 02:56:20.381402 (+ 1947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:56:20.381409 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:56:20.381498 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:56:20.381590 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:56:20.383419 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:56:20.384192 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:56:20.385855 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:56:20.386357 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:56:20.386475 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:56:20.390297 (+ 3822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:56:20.390303 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:56:20.390307 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:56:20.390343 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:56:20.393649 (+ 3306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:56:20.394671 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:56:20.394676 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:56:20.394678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:56:20.394826 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:56:20.395298 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:56:20.395302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:56:20.395304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:56:20.395667 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:56:20.395828 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 02:56:20.396566 (+ 738us) server_negotiation.cc:299] Negotiation successful
0806 02:56:20.396853 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 02:56:20.398355 18476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:20.378514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:40273 (local address 127.25.211.130:56015)
0806 02:56:20.379280 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:20.379322 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 02:56:20.381093 (+ 1771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:56:20.381891 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:56:20.381900 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:56:20.382558 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:56:20.383232 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:56:20.383251 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:56:20.384359 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:56:20.384363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:56:20.385699 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:56:20.385713 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:56:20.386610 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:56:20.386617 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:56:20.386862 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:56:20.387529 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:56:20.387555 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:56:20.390003 (+ 2448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:56:20.393867 (+ 3864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:56:20.393877 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:56:20.393895 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:56:20.394528 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:56:20.394944 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:56:20.394949 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:56:20.394952 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:56:20.395162 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:56:20.395808 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:56:20.395816 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:56:20.396174 (+ 358us) client_negotiation.cc:769] Sending connection context
0806 02:56:20.396432 (+ 258us) client_negotiation.cc:240] Negotiation successful
0806 02:56:20.397002 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":305,"threads_started":1}
I20260806 02:56:20.399869 18472 heartbeater.cc:347] Connected to a master server at 127.25.211.190:40273
I20260806 02:56:20.400206 18472 heartbeater.cc:464] Registering TS with master...
I20260806 02:56:20.401010 18472 heartbeater.cc:511] Master 127.25.211.190:40273 requested a full tablet report, sending...
I20260806 02:56:20.402583 17993 ts_manager.cc:194] Registered new tserver with Master: ad3f7123cd8d4dcebd11197c8bfd5d39 (127.25.211.130:42087)
I20260806 02:56:20.403434 17993 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.211.130@KRBTEST.COM'} at 127.25.211.130:56015
W20260806 02:56:20.741082 18477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:56:20.741393 18477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:56:20.741513 18477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:56:20.749130 18477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:56:20.749260 18477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:56:20.749413 18477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.131
I20260806 02:56:20.759655 18477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:37079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab
--principal=kudu/127.25.211.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.131:43105
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.211.131
--webserver_port=40627
--tserver_master_addrs=127.25.211.190:40273
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.131
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:56:20.761006 18477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:56:20.762383 18477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:56:20.775902 18483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:56:20.775933 18485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:56:20.775840 18488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:56:20.776742 18477 server_base.cc:1034] running on GCE node
I20260806 02:56:20.777316 18477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:56:20.778312 18477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:56:20.779524 18477 hybrid_clock.cc:648] HybridClock initialized: now 1785984980779457 us; error 58 us; skew 500 ppm
I20260806 02:56:20.779852 18477 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:56:20 dist-test-slave-zj8q krb5kdc[17141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984980, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:56:20.786832 18477 init.cc:377] Logged in from keytab as kudu/127.25.211.131@KRBTEST.COM (short username kudu)
I20260806 02:56:20.788403 18477 webserver.cc:460] Webserver started at http://127.25.211.131:40627/ using document root <none> and password file <none>
I20260806 02:56:20.789218 18477 fs_manager.cc:359] Metadata directory not provided
I20260806 02:56:20.789397 18477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:56:20.795042 18477 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 02:56:20.797849 18501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:56:20.799294 18477 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 02:56:20.799463 18477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-2/wal
uuid: "6b00a6aadf6240b0b34a3c2594e2e0d7"
format_stamp: "Formatted at 2026-08-06 02:55:57 on dist-test-slave-zj8q"
server_key: "a856e286e26af2352781b931775f42e6"
server_key_iv: "772930b073f49b4447b74462bb61c8d3"
server_key_version: "encryptionkey@0"
I20260806 02:56:20.800458 18477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:56:20.820672 18477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:56:20.825018 18477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:56:20.825304 18477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:56:20.826421 18477 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:56:20.827867 18477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:56:20.827978 18477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:56:20.828091 18477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:56:20.828163 18477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:56:20.860630 18477 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.131:43105
I20260806 02:56:20.860718 18679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.131:43105 every 8 connection(s)
I20260806 02:56:20.862354 18477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
I20260806 02:56:20.870277 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 18477
Aug 06 02:56:20 dist-test-slave-zj8q krb5kdc[17141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785984980, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.131@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:56:20.884136 18275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:20.865105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:33413 (local address 127.25.211.190:40273)
0806 02:56:20.865321 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0806 02:56:20.865326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:56:20.866548 (+ 1222us) server_negotiation.cc:407] Connection header received
0806 02:56:20.868341 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:56:20.868345 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:56:20.868413 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:56:20.868500 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:56:20.870086 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:56:20.870985 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:56:20.872894 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:56:20.873364 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:56:20.873496 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:56:20.877798 (+ 4302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:56:20.877804 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:56:20.877810 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:56:20.877856 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:56:20.880837 (+ 2981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:56:20.881594 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:56:20.881599 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:56:20.881601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:56:20.881732 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:56:20.882175 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:56:20.882180 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:56:20.882182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:56:20.882506 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:56:20.882684 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0806 02:56:20.883519 (+ 835us) server_negotiation.cc:299] Negotiation successful
0806 02:56:20.883753 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260806 02:56:20.885059 18684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:20.865588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:40273 (local address 127.25.211.131:33413)
0806 02:56:20.866378 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:20.866424 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 02:56:20.868101 (+ 1677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:56:20.868733 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:56:20.868741 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:56:20.869261 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:56:20.869919 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:56:20.869946 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:56:20.871163 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:56:20.871167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:56:20.872743 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:56:20.872757 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:56:20.873618 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:56:20.873626 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:56:20.873894 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:56:20.874663 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:56:20.874701 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:56:20.877545 (+ 2844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:56:20.880995 (+ 3450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:56:20.881004 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:56:20.881020 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:56:20.881474 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:56:20.881841 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:56:20.881844 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:56:20.881846 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:56:20.882045 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:56:20.882670 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:56:20.882677 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:56:20.883076 (+ 399us) client_negotiation.cc:769] Sending connection context
0806 02:56:20.883395 (+ 319us) client_negotiation.cc:240] Negotiation successful
0806 02:56:20.883845 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":307,"threads_started":1}
I20260806 02:56:20.886636 18681 heartbeater.cc:347] Connected to a master server at 127.25.211.190:40273
I20260806 02:56:20.886970 18681 heartbeater.cc:464] Registering TS with master...
I20260806 02:56:20.887730 18681 heartbeater.cc:511] Master 127.25.211.190:40273 requested a full tablet report, sending...
I20260806 02:56:20.888259 18686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:20.871308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34396 (local address 127.25.211.190:40273)
0806 02:56:20.873620 (+ 2312us) server_negotiation.cc:206] Beginning negotiation
0806 02:56:20.873627 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:56:20.873642 (+ 15us) server_negotiation.cc:407] Connection header received
0806 02:56:20.873703 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:56:20.873707 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:56:20.873774 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:56:20.873856 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:56:20.875979 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:56:20.877160 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:56:20.879066 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:56:20.879549 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:56:20.879644 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:56:20.882501 (+ 2857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:56:20.882506 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:56:20.882511 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:56:20.882541 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:56:20.885462 (+ 2921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:56:20.886249 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:56:20.886253 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:56:20.886255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:56:20.886375 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:56:20.886770 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:56:20.886775 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:56:20.886778 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:56:20.887076 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:56:20.887330 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0806 02:56:20.887668 (+ 338us) server_negotiation.cc:299] Negotiation successful
0806 02:56:20.887910 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2209,"thread_start_us":2054,"threads_started":1}
I20260806 02:56:20.889317 17993 ts_manager.cc:194] Registered new tserver with Master: 6b00a6aadf6240b0b34a3c2594e2e0d7 (127.25.211.131:43105)
I20260806 02:56:20.890283 17993 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.211.131@KRBTEST.COM'} at 127.25.211.131:33413
I20260806 02:56:20.891110 26446 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:56:20.897874 18686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:20.892099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34398 (local address 127.25.211.190:40273)
0806 02:56:20.892311 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0806 02:56:20.892318 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:56:20.892806 (+ 488us) server_negotiation.cc:407] Connection header received
0806 02:56:20.893056 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:56:20.893062 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:56:20.893205 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:56:20.893324 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:56:20.893734 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:56:20.894500 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:56:20.895960 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:56:20.896376 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:56:20.896463 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:56:20.897035 (+ 572us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:56:20.897335 (+ 300us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0806 02:56:20.897636 (+ 301us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34398: token expired
Metrics: {"server-negotiator.queue_time_us":62}
W20260806 02:56:20.898010 18686 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34398: token expired
W20260806 02:56:20.898682 17874 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.25.211.190:40273)
I20260806 02:56:20.911972 18686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:20.899157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34410 (local address 127.25.211.190:40273)
0806 02:56:20.899378 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0806 02:56:20.899382 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:56:20.899444 (+ 62us) server_negotiation.cc:407] Connection header received
0806 02:56:20.899584 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:56:20.899587 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:56:20.899646 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:56:20.899721 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:56:20.901074 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:56:20.901805 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:56:20.903195 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:56:20.903614 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:56:20.903736 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:56:20.906726 (+ 2990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:56:20.906731 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:56:20.906734 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:56:20.906768 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:56:20.909118 (+ 2350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:56:20.909877 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:56:20.909883 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:56:20.909887 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:56:20.910026 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:56:20.910484 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:56:20.910489 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:56:20.910493 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:56:20.910841 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:56:20.911071 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0806 02:56:20.911409 (+ 338us) server_negotiation.cc:299] Negotiation successful
0806 02:56:20.911637 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260806 02:56:20.915665 17993 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34410:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:56:20.923329 17993 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:56:20.938829 18273 heartbeater.cc:503] Master 127.25.211.190:40273 was elected leader, sending a full tablet report...
I20260806 02:56:20.946861 18696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:20.935656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34552 (local address 127.25.211.130:42087)
0806 02:56:20.936288 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0806 02:56:20.936299 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:56:20.937862 (+ 1563us) server_negotiation.cc:407] Connection header received
0806 02:56:20.938359 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:56:20.938368 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:56:20.938612 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:56:20.938768 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:56:20.939833 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:56:20.941321 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:56:20.943935 (+ 2614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:56:20.945552 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:56:20.945720 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:56:20.945903 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0806 02:56:20.946351 (+ 448us) server_negotiation.cc:299] Negotiation successful
0806 02:56:20.946531 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":378,"threads_started":1}
I20260806 02:56:20.947057 18698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:20.935240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.130:42087 (local address 127.0.0.1:34552)
0806 02:56:20.937719 (+ 2479us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:20.937767 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 02:56:20.938035 (+ 268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:56:20.939097 (+ 1062us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:56:20.939105 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:56:20.939134 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:56:20.939593 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:56:20.939607 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:56:20.941511 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:56:20.941516 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:56:20.943740 (+ 2224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:56:20.943759 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:56:20.945801 (+ 2042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:56:20.945808 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:56:20.946009 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:56:20.946063 (+ 54us) client_negotiation.cc:769] Sending connection context
0806 02:56:20.946281 (+ 218us) client_negotiation.cc:240] Negotiation successful
0806 02:56:20.946487 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2243,"mutex_wait_us":24,"spinlock_wait_cycles":2304,"thread_start_us":2708,"threads_started":1}
I20260806 02:56:20.947717 18695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:20.935422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.131:43105 (local address 127.0.0.1:38924)
0806 02:56:20.938246 (+ 2824us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:20.938262 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:56:20.938373 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:56:20.939271 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:56:20.939276 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:56:20.939315 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:56:20.940717 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:56:20.940747 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:56:20.942620 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:56:20.942626 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:56:20.944750 (+ 2124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:56:20.944766 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:56:20.946857 (+ 2091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:56:20.946864 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:56:20.947049 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:56:20.947080 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 02:56:20.947210 (+ 130us) client_negotiation.cc:240] Negotiation successful
0806 02:56:20.947340 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2686,"mutex_wait_us":1,"spinlock_wait_cycles":1280,"thread_start_us":1929,"threads_started":1}
I20260806 02:56:20.948422 18701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:20.937449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:41493 (local address 127.0.0.1:45348)
0806 02:56:20.940253 (+ 2804us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:20.940270 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:56:20.940468 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:56:20.941425 (+ 957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:56:20.941428 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:56:20.941448 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:56:20.941773 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:56:20.941781 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:56:20.943332 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:56:20.943338 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:56:20.945270 (+ 1932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:56:20.945281 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:56:20.947588 (+ 2307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:56:20.947595 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:56:20.947791 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:56:20.947823 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 02:56:20.947914 (+ 91us) client_negotiation.cc:240] Negotiation successful
0806 02:56:20.948032 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2673,"mutex_wait_us":668,"thread_start_us":1806,"threads_started":1}
I20260806 02:56:20.948490 18700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:20.937941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45348 (local address 127.25.211.129:41493)
0806 02:56:20.939763 (+ 1822us) server_negotiation.cc:206] Beginning negotiation
0806 02:56:20.939772 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:56:20.940362 (+ 590us) server_negotiation.cc:407] Connection header received
0806 02:56:20.940663 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:56:20.940671 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:56:20.940898 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:56:20.941055 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:56:20.941925 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:56:20.943191 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:56:20.945410 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:56:20.946817 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:56:20.947008 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:56:20.947166 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0806 02:56:20.947973 (+ 807us) server_negotiation.cc:299] Negotiation successful
0806 02:56:20.948149 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":421,"threads_started":1}
I20260806 02:56:20.949307 18697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:20.935752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38924 (local address 127.25.211.131:43105)
0806 02:56:20.936930 (+ 1178us) server_negotiation.cc:206] Beginning negotiation
0806 02:56:20.936939 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:56:20.938500 (+ 1561us) server_negotiation.cc:407] Connection header received
0806 02:56:20.938612 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:56:20.938619 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:56:20.938822 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:56:20.938994 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:56:20.940936 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:56:20.942453 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:56:20.944935 (+ 2482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:56:20.946271 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:56:20.946465 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:56:20.946624 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0806 02:56:20.948803 (+ 2179us) server_negotiation.cc:299] Negotiation successful
0806 02:56:20.948974 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":996,"thread_start_us":831,"threads_started":1}
I20260806 02:56:20.951397 18376 tablet_service.cc:1467] Processing CreateTablet for tablet 67e2461daa194fda98eb915a36f99209 (DEFAULT_TABLE table=test-table [id=805137b10abd4fc790fe63baa1290980]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:56:20.952018 18163 tablet_service.cc:1467] Processing CreateTablet for tablet 67e2461daa194fda98eb915a36f99209 (DEFAULT_TABLE table=test-table [id=805137b10abd4fc790fe63baa1290980]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:56:20.952786 18376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67e2461daa194fda98eb915a36f99209. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:56:20.953558 18163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67e2461daa194fda98eb915a36f99209. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:56:20.953143 18574 tablet_service.cc:1467] Processing CreateTablet for tablet 67e2461daa194fda98eb915a36f99209 (DEFAULT_TABLE table=test-table [id=805137b10abd4fc790fe63baa1290980]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:56:20.954530 18574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67e2461daa194fda98eb915a36f99209. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:56:20.977115 18705 tablet_bootstrap.cc:492] T 67e2461daa194fda98eb915a36f99209 P e510bcdf18a54a4392a3392d53e84d4b: Bootstrap starting.
I20260806 02:56:20.978508 18702 tablet_bootstrap.cc:492] T 67e2461daa194fda98eb915a36f99209 P ad3f7123cd8d4dcebd11197c8bfd5d39: Bootstrap starting.
I20260806 02:56:20.981477 18705 tablet_bootstrap.cc:654] T 67e2461daa194fda98eb915a36f99209 P e510bcdf18a54a4392a3392d53e84d4b: Neither blocks nor log segments found. Creating new log.
I20260806 02:56:20.981611 18702 tablet_bootstrap.cc:654] T 67e2461daa194fda98eb915a36f99209 P ad3f7123cd8d4dcebd11197c8bfd5d39: Neither blocks nor log segments found. Creating new log.
I20260806 02:56:20.982978 18705 log.cc:824] T 67e2461daa194fda98eb915a36f99209 P e510bcdf18a54a4392a3392d53e84d4b: Log is configured to *not* fsync() on all Append() calls
I20260806 02:56:20.983471 18704 tablet_bootstrap.cc:492] T 67e2461daa194fda98eb915a36f99209 P 6b00a6aadf6240b0b34a3c2594e2e0d7: Bootstrap starting.
I20260806 02:56:20.984628 18702 log.cc:824] T 67e2461daa194fda98eb915a36f99209 P ad3f7123cd8d4dcebd11197c8bfd5d39: Log is configured to *not* fsync() on all Append() calls
I20260806 02:56:20.986944 18704 tablet_bootstrap.cc:654] T 67e2461daa194fda98eb915a36f99209 P 6b00a6aadf6240b0b34a3c2594e2e0d7: Neither blocks nor log segments found. Creating new log.
I20260806 02:56:20.987941 18702 tablet_bootstrap.cc:492] T 67e2461daa194fda98eb915a36f99209 P ad3f7123cd8d4dcebd11197c8bfd5d39: No bootstrap required, opened a new log
I20260806 02:56:20.988217 18702 ts_tablet_manager.cc:1397] T 67e2461daa194fda98eb915a36f99209 P ad3f7123cd8d4dcebd11197c8bfd5d39: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260806 02:56:20.988286 18705 tablet_bootstrap.cc:492] T 67e2461daa194fda98eb915a36f99209 P e510bcdf18a54a4392a3392d53e84d4b: No bootstrap required, opened a new log
I20260806 02:56:20.988348 18704 log.cc:824] T 67e2461daa194fda98eb915a36f99209 P 6b00a6aadf6240b0b34a3c2594e2e0d7: Log is configured to *not* fsync() on all Append() calls
I20260806 02:56:20.988647 18705 ts_tablet_manager.cc:1397] T 67e2461daa194fda98eb915a36f99209 P e510bcdf18a54a4392a3392d53e84d4b: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20260806 02:56:20.991744 18704 tablet_bootstrap.cc:492] T 67e2461daa194fda98eb915a36f99209 P 6b00a6aadf6240b0b34a3c2594e2e0d7: No bootstrap required, opened a new log
I20260806 02:56:20.992067 18704 ts_tablet_manager.cc:1397] T 67e2461daa194fda98eb915a36f99209 P 6b00a6aadf6240b0b34a3c2594e2e0d7: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.005s
I20260806 02:56:20.996368 18702 raft_consensus.cc:348] T 67e2461daa194fda98eb915a36f99209 P ad3f7123cd8d4dcebd11197c8bfd5d39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3f7123cd8d4dcebd11197c8bfd5d39" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 42087 } } peers { permanent_uuid: "6b00a6aadf6240b0b34a3c2594e2e0d7" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 43105 } } peers { permanent_uuid: "e510bcdf18a54a4392a3392d53e84d4b" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 41493 } }
I20260806 02:56:20.996788 18702 raft_consensus.cc:374] T 67e2461daa194fda98eb915a36f99209 P ad3f7123cd8d4dcebd11197c8bfd5d39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:56:20.996913 18702 raft_consensus.cc:729] T 67e2461daa194fda98eb915a36f99209 P ad3f7123cd8d4dcebd11197c8bfd5d39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad3f7123cd8d4dcebd11197c8bfd5d39, State: Initialized, Role: FOLLOWER
I20260806 02:56:20.997524 18702 consensus_queue.cc:260] T 67e2461daa194fda98eb915a36f99209 P ad3f7123cd8d4dcebd11197c8bfd5d39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3f7123cd8d4dcebd11197c8bfd5d39" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 42087 } } peers { permanent_uuid: "6b00a6aadf6240b0b34a3c2594e2e0d7" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 43105 } } peers { permanent_uuid: "e510bcdf18a54a4392a3392d53e84d4b" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 41493 } }
I20260806 02:56:20.998728 18472 heartbeater.cc:503] Master 127.25.211.190:40273 was elected leader, sending a full tablet report...
I20260806 02:56:20.999010 18705 raft_consensus.cc:348] T 67e2461daa194fda98eb915a36f99209 P e510bcdf18a54a4392a3392d53e84d4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3f7123cd8d4dcebd11197c8bfd5d39" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 42087 } } peers { permanent_uuid: "6b00a6aadf6240b0b34a3c2594e2e0d7" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 43105 } } peers { permanent_uuid: "e510bcdf18a54a4392a3392d53e84d4b" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 41493 } }
I20260806 02:56:20.999498 18705 raft_consensus.cc:374] T 67e2461daa194fda98eb915a36f99209 P e510bcdf18a54a4392a3392d53e84d4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:56:20.999603 18705 raft_consensus.cc:729] T 67e2461daa194fda98eb915a36f99209 P e510bcdf18a54a4392a3392d53e84d4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e510bcdf18a54a4392a3392d53e84d4b, State: Initialized, Role: FOLLOWER
I20260806 02:56:21.000377 18705 consensus_queue.cc:260] T 67e2461daa194fda98eb915a36f99209 P e510bcdf18a54a4392a3392d53e84d4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3f7123cd8d4dcebd11197c8bfd5d39" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 42087 } } peers { permanent_uuid: "6b00a6aadf6240b0b34a3c2594e2e0d7" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 43105 } } peers { permanent_uuid: "e510bcdf18a54a4392a3392d53e84d4b" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 41493 } }
I20260806 02:56:21.000679 18702 ts_tablet_manager.cc:1428] T 67e2461daa194fda98eb915a36f99209 P ad3f7123cd8d4dcebd11197c8bfd5d39: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260806 02:56:21.002279 18705 ts_tablet_manager.cc:1428] T 67e2461daa194fda98eb915a36f99209 P e510bcdf18a54a4392a3392d53e84d4b: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260806 02:56:21.006024 18704 raft_consensus.cc:348] T 67e2461daa194fda98eb915a36f99209 P 6b00a6aadf6240b0b34a3c2594e2e0d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3f7123cd8d4dcebd11197c8bfd5d39" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 42087 } } peers { permanent_uuid: "6b00a6aadf6240b0b34a3c2594e2e0d7" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 43105 } } peers { permanent_uuid: "e510bcdf18a54a4392a3392d53e84d4b" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 41493 } }
I20260806 02:56:21.006507 18704 raft_consensus.cc:374] T 67e2461daa194fda98eb915a36f99209 P 6b00a6aadf6240b0b34a3c2594e2e0d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:56:21.006615 18704 raft_consensus.cc:729] T 67e2461daa194fda98eb915a36f99209 P 6b00a6aadf6240b0b34a3c2594e2e0d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b00a6aadf6240b0b34a3c2594e2e0d7, State: Initialized, Role: FOLLOWER
I20260806 02:56:21.007270 18704 consensus_queue.cc:260] T 67e2461daa194fda98eb915a36f99209 P 6b00a6aadf6240b0b34a3c2594e2e0d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3f7123cd8d4dcebd11197c8bfd5d39" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 42087 } } peers { permanent_uuid: "6b00a6aadf6240b0b34a3c2594e2e0d7" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 43105 } } peers { permanent_uuid: "e510bcdf18a54a4392a3392d53e84d4b" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 41493 } }
I20260806 02:56:21.008353 18681 heartbeater.cc:503] Master 127.25.211.190:40273 was elected leader, sending a full tablet report...
I20260806 02:56:21.008878 18704 ts_tablet_manager.cc:1428] T 67e2461daa194fda98eb915a36f99209 P 6b00a6aadf6240b0b34a3c2594e2e0d7: Time spent starting tablet: real 0.017s user 0.014s sys 0.000s
I20260806 02:56:21.081786 18714 raft_consensus.cc:482] T 67e2461daa194fda98eb915a36f99209 P 6b00a6aadf6240b0b34a3c2594e2e0d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:56:21.082123 18714 raft_consensus.cc:504] T 67e2461daa194fda98eb915a36f99209 P 6b00a6aadf6240b0b34a3c2594e2e0d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3f7123cd8d4dcebd11197c8bfd5d39" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 42087 } } peers { permanent_uuid: "6b00a6aadf6240b0b34a3c2594e2e0d7" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 43105 } } peers { permanent_uuid: "e510bcdf18a54a4392a3392d53e84d4b" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 41493 } }
I20260806 02:56:21.083922 18714 leader_election.cc:288] T 67e2461daa194fda98eb915a36f99209 P 6b00a6aadf6240b0b34a3c2594e2e0d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad3f7123cd8d4dcebd11197c8bfd5d39 (127.25.211.130:42087), e510bcdf18a54a4392a3392d53e84d4b (127.25.211.129:41493)
I20260806 02:56:21.090284 18700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:21.084530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:41989 (local address 127.25.211.129:41493)
0806 02:56:21.084745 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0806 02:56:21.084750 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:56:21.084770 (+ 20us) server_negotiation.cc:407] Connection header received
0806 02:56:21.084920 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:56:21.084924 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:56:21.084983 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:56:21.085058 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:56:21.085486 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:56:21.086492 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:56:21.088318 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:56:21.089338 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:56:21.089461 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:56:21.089518 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0806 02:56:21.089895 (+ 377us) server_negotiation.cc:299] Negotiation successful
0806 02:56:21.090005 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 02:56:21.090312 18684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:21.084248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:41493 (local address 127.25.211.131:41989)
0806 02:56:21.084623 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:21.084643 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 02:56:21.084776 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:56:21.085103 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:56:21.085108 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:56:21.085134 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:56:21.085329 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:56:21.085342 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:56:21.086634 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:56:21.086640 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:56:21.088144 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:56:21.088157 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:56:21.089495 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:56:21.089501 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:56:21.089693 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:56:21.089731 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 02:56:21.089829 (+ 98us) client_negotiation.cc:240] Negotiation successful
0806 02:56:21.089972 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225}
I20260806 02:56:21.090965 18696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:21.084431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:60579 (local address 127.25.211.130:42087)
0806 02:56:21.084684 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0806 02:56:21.084690 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:56:21.085312 (+ 622us) server_negotiation.cc:407] Connection header received
0806 02:56:21.085451 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:56:21.085458 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:56:21.085535 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:56:21.085602 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:56:21.086203 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:56:21.087093 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:56:21.089018 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:56:21.090018 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:56:21.090184 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:56:21.090252 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 02:56:21.090650 (+ 398us) server_negotiation.cc:299] Negotiation successful
0806 02:56:21.090778 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 02:56:21.091528 18715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:21.084389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.130:42087 (local address 127.25.211.131:60579)
0806 02:56:21.085201 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:21.085220 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:56:21.085334 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:56:21.085653 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:56:21.085657 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:56:21.085682 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:56:21.086035 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:56:21.086048 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:56:21.087252 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:56:21.087260 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:56:21.088855 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:56:21.088871 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:56:21.090278 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:56:21.090285 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:56:21.090461 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:56:21.090493 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 02:56:21.091183 (+ 690us) client_negotiation.cc:240] Negotiation successful
0806 02:56:21.091287 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":694,"thread_start_us":445,"threads_started":1}
I20260806 02:56:21.091379 18197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e2461daa194fda98eb915a36f99209" candidate_uuid: "6b00a6aadf6240b0b34a3c2594e2e0d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e510bcdf18a54a4392a3392d53e84d4b" is_pre_election: true
I20260806 02:56:21.091904 18197 raft_consensus.cc:2454] T 67e2461daa194fda98eb915a36f99209 P e510bcdf18a54a4392a3392d53e84d4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b00a6aadf6240b0b34a3c2594e2e0d7 in term 0.
I20260806 02:56:21.092633 18503 leader_election.cc:302] T 67e2461daa194fda98eb915a36f99209 P 6b00a6aadf6240b0b34a3c2594e2e0d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b00a6aadf6240b0b34a3c2594e2e0d7, e510bcdf18a54a4392a3392d53e84d4b; no voters:
I20260806 02:56:21.092581 18400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e2461daa194fda98eb915a36f99209" candidate_uuid: "6b00a6aadf6240b0b34a3c2594e2e0d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad3f7123cd8d4dcebd11197c8bfd5d39" is_pre_election: true
I20260806 02:56:21.093093 18714 raft_consensus.cc:2781] T 67e2461daa194fda98eb915a36f99209 P 6b00a6aadf6240b0b34a3c2594e2e0d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:56:21.093119 18400 raft_consensus.cc:2454] T 67e2461daa194fda98eb915a36f99209 P ad3f7123cd8d4dcebd11197c8bfd5d39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b00a6aadf6240b0b34a3c2594e2e0d7 in term 0.
I20260806 02:56:21.093272 18714 raft_consensus.cc:482] T 67e2461daa194fda98eb915a36f99209 P 6b00a6aadf6240b0b34a3c2594e2e0d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:56:21.093385 18714 raft_consensus.cc:3037] T 67e2461daa194fda98eb915a36f99209 P 6b00a6aadf6240b0b34a3c2594e2e0d7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:56:21.095796 18714 raft_consensus.cc:504] T 67e2461daa194fda98eb915a36f99209 P 6b00a6aadf6240b0b34a3c2594e2e0d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3f7123cd8d4dcebd11197c8bfd5d39" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 42087 } } peers { permanent_uuid: "6b00a6aadf6240b0b34a3c2594e2e0d7" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 43105 } } peers { permanent_uuid: "e510bcdf18a54a4392a3392d53e84d4b" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 41493 } }
I20260806 02:56:21.096437 18714 leader_election.cc:288] T 67e2461daa194fda98eb915a36f99209 P 6b00a6aadf6240b0b34a3c2594e2e0d7 [CANDIDATE]: Term 1 election: Requested vote from peers ad3f7123cd8d4dcebd11197c8bfd5d39 (127.25.211.130:42087), e510bcdf18a54a4392a3392d53e84d4b (127.25.211.129:41493)
I20260806 02:56:21.096951 18197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e2461daa194fda98eb915a36f99209" candidate_uuid: "6b00a6aadf6240b0b34a3c2594e2e0d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e510bcdf18a54a4392a3392d53e84d4b"
I20260806 02:56:21.096940 18400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e2461daa194fda98eb915a36f99209" candidate_uuid: "6b00a6aadf6240b0b34a3c2594e2e0d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad3f7123cd8d4dcebd11197c8bfd5d39"
I20260806 02:56:21.097172 18400 raft_consensus.cc:3037] T 67e2461daa194fda98eb915a36f99209 P ad3f7123cd8d4dcebd11197c8bfd5d39 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:56:21.097175 18197 raft_consensus.cc:3037] T 67e2461daa194fda98eb915a36f99209 P e510bcdf18a54a4392a3392d53e84d4b [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:56:21.099654 18400 raft_consensus.cc:2454] T 67e2461daa194fda98eb915a36f99209 P ad3f7123cd8d4dcebd11197c8bfd5d39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b00a6aadf6240b0b34a3c2594e2e0d7 in term 1.
I20260806 02:56:21.099655 18197 raft_consensus.cc:2454] T 67e2461daa194fda98eb915a36f99209 P e510bcdf18a54a4392a3392d53e84d4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b00a6aadf6240b0b34a3c2594e2e0d7 in term 1.
I20260806 02:56:21.100076 18507 leader_election.cc:302] T 67e2461daa194fda98eb915a36f99209 P 6b00a6aadf6240b0b34a3c2594e2e0d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b00a6aadf6240b0b34a3c2594e2e0d7, ad3f7123cd8d4dcebd11197c8bfd5d39; no voters:
I20260806 02:56:21.100433 18714 raft_consensus.cc:2781] T 67e2461daa194fda98eb915a36f99209 P 6b00a6aadf6240b0b34a3c2594e2e0d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:56:21.100924 18714 raft_consensus.cc:686] T 67e2461daa194fda98eb915a36f99209 P 6b00a6aadf6240b0b34a3c2594e2e0d7 [term 1 LEADER]: Becoming Leader. State: Replica: 6b00a6aadf6240b0b34a3c2594e2e0d7, State: Running, Role: LEADER
I20260806 02:56:21.101408 18714 consensus_queue.cc:237] T 67e2461daa194fda98eb915a36f99209 P 6b00a6aadf6240b0b34a3c2594e2e0d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3f7123cd8d4dcebd11197c8bfd5d39" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 42087 } } peers { permanent_uuid: "6b00a6aadf6240b0b34a3c2594e2e0d7" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 43105 } } peers { permanent_uuid: "e510bcdf18a54a4392a3392d53e84d4b" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 41493 } }
I20260806 02:56:21.106441 17993 catalog_manager.cc:5495] T 67e2461daa194fda98eb915a36f99209 P 6b00a6aadf6240b0b34a3c2594e2e0d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b00a6aadf6240b0b34a3c2594e2e0d7 (127.25.211.131). New cstate: current_term: 1 leader_uuid: "6b00a6aadf6240b0b34a3c2594e2e0d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3f7123cd8d4dcebd11197c8bfd5d39" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 42087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b00a6aadf6240b0b34a3c2594e2e0d7" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 43105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e510bcdf18a54a4392a3392d53e84d4b" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 41493 } health_report { overall_health: UNKNOWN } } }
W20260806 02:56:21.114023 18682 tablet.cc:2369] T 67e2461daa194fda98eb915a36f99209 P 6b00a6aadf6240b0b34a3c2594e2e0d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:56:21.127243 18474 tablet.cc:2369] T 67e2461daa194fda98eb915a36f99209 P ad3f7123cd8d4dcebd11197c8bfd5d39: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:56:21.146517 18697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:21.141394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38932 (local address 127.25.211.131:43105)
0806 02:56:21.141599 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0806 02:56:21.141604 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:56:21.141622 (+ 18us) server_negotiation.cc:407] Connection header received
0806 02:56:21.141795 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:56:21.141799 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:56:21.141877 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:56:21.142003 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:56:21.142379 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:56:21.143100 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:56:21.144394 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:56:21.144799 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:56:21.144887 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:56:21.145385 (+ 498us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:56:21.145930 (+ 545us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:56:21.146065 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0806 02:56:21.146132 (+ 67us) server_negotiation.cc:299] Negotiation successful
0806 02:56:21.146238 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 02:56:21.155385 18400 raft_consensus.cc:1261] T 67e2461daa194fda98eb915a36f99209 P ad3f7123cd8d4dcebd11197c8bfd5d39 [term 1 FOLLOWER]: Refusing update from remote peer 6b00a6aadf6240b0b34a3c2594e2e0d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260806 02:56:21.155362 18274 tablet.cc:2369] T 67e2461daa194fda98eb915a36f99209 P e510bcdf18a54a4392a3392d53e84d4b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:56:21.155442 18197 raft_consensus.cc:1261] T 67e2461daa194fda98eb915a36f99209 P e510bcdf18a54a4392a3392d53e84d4b [term 1 FOLLOWER]: Refusing update from remote peer 6b00a6aadf6240b0b34a3c2594e2e0d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:56:21.156409 18714 consensus_queue.cc:1035] T 67e2461daa194fda98eb915a36f99209 P 6b00a6aadf6240b0b34a3c2594e2e0d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e510bcdf18a54a4392a3392d53e84d4b" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 41493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:56:21.156760 18718 consensus_queue.cc:1035] T 67e2461daa194fda98eb915a36f99209 P 6b00a6aadf6240b0b34a3c2594e2e0d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad3f7123cd8d4dcebd11197c8bfd5d39" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 42087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:56:21.170912 18721 mvcc.cc:205] Tried to move back new op lower bound from 7315394482801098752 to 7315394482595463168. Current Snapshot: MvccSnapshot[applied={T|T < 7315394482801098752}]
I20260806 02:56:21.186528 17992 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:56:21.186846 17992 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:56:21.190645 17992 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9: Sending DeleteTablet for 3 replicas of tablet 67e2461daa194fda98eb915a36f99209
I20260806 02:56:21.191928 18574 tablet_service.cc:1514] Processing DeleteTablet for tablet 67e2461daa194fda98eb915a36f99209 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:56:21 UTC) from {username='kudu', principal='kudu/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:38924
I20260806 02:56:21.191923 18163 tablet_service.cc:1514] Processing DeleteTablet for tablet 67e2461daa194fda98eb915a36f99209 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:56:21 UTC) from {username='kudu', principal='kudu/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:45348
I20260806 02:56:21.191947 18376 tablet_service.cc:1514] Processing DeleteTablet for tablet 67e2461daa194fda98eb915a36f99209 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:56:21 UTC) from {username='kudu', principal='kudu/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:34552
I20260806 02:56:21.194700 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 18042
I20260806 02:56:21.204478 18739 tablet_replica.cc:330] T 67e2461daa194fda98eb915a36f99209 P 6b00a6aadf6240b0b34a3c2594e2e0d7: stopping tablet replica
I20260806 02:56:21.205297 18738 tablet_replica.cc:330] T 67e2461daa194fda98eb915a36f99209 P e510bcdf18a54a4392a3392d53e84d4b: stopping tablet replica
I20260806 02:56:21.205448 18740 tablet_replica.cc:330] T 67e2461daa194fda98eb915a36f99209 P ad3f7123cd8d4dcebd11197c8bfd5d39: stopping tablet replica
I20260806 02:56:21.205919 18738 raft_consensus.cc:2229] T 67e2461daa194fda98eb915a36f99209 P e510bcdf18a54a4392a3392d53e84d4b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:56:21.205919 18740 raft_consensus.cc:2229] T 67e2461daa194fda98eb915a36f99209 P ad3f7123cd8d4dcebd11197c8bfd5d39 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:56:21.206560 18739 raft_consensus.cc:2229] T 67e2461daa194fda98eb915a36f99209 P 6b00a6aadf6240b0b34a3c2594e2e0d7 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:56:21.206718 18740 raft_consensus.cc:2258] T 67e2461daa194fda98eb915a36f99209 P ad3f7123cd8d4dcebd11197c8bfd5d39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:56:21.207078 18738 raft_consensus.cc:2258] T 67e2461daa194fda98eb915a36f99209 P e510bcdf18a54a4392a3392d53e84d4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:56:21.207468 18739 raft_consensus.cc:2258] T 67e2461daa194fda98eb915a36f99209 P 6b00a6aadf6240b0b34a3c2594e2e0d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:56:21.210032 18739 ts_tablet_manager.cc:1905] T 67e2461daa194fda98eb915a36f99209 P 6b00a6aadf6240b0b34a3c2594e2e0d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:56:21.210814 18738 ts_tablet_manager.cc:1905] T 67e2461daa194fda98eb915a36f99209 P e510bcdf18a54a4392a3392d53e84d4b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:56:21.213366 18740 ts_tablet_manager.cc:1905] T 67e2461daa194fda98eb915a36f99209 P ad3f7123cd8d4dcebd11197c8bfd5d39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:56:21.214286 18700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:21.196007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45352 (local address 127.25.211.129:41493)
0806 02:56:21.196198 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0806 02:56:21.196204 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:56:21.196225 (+ 21us) server_negotiation.cc:407] Connection header received
0806 02:56:21.196289 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:56:21.196294 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:56:21.196381 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:56:21.196492 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:56:21.198583 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:56:21.199843 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:56:21.202023 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:56:21.202753 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:56:21.202905 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:56:21.206828 (+ 3923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:56:21.206842 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:56:21.206860 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:56:21.206903 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:56:21.210847 (+ 3944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:56:21.211819 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:56:21.211829 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:56:21.211832 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:56:21.212011 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:56:21.212472 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:56:21.212477 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:56:21.212480 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:56:21.212996 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:56:21.213311 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0806 02:56:21.213667 (+ 356us) server_negotiation.cc:299] Negotiation successful
0806 02:56:21.213904 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 02:56:21.215211 18262 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:56:21.219658 18739 ts_tablet_manager.cc:1918] T 67e2461daa194fda98eb915a36f99209 P 6b00a6aadf6240b0b34a3c2594e2e0d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:56:21.219965 18739 log.cc:1196] T 67e2461daa194fda98eb915a36f99209 P 6b00a6aadf6240b0b34a3c2594e2e0d7: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-2/wal/wals/67e2461daa194fda98eb915a36f99209
I20260806 02:56:21.220649 18739 ts_tablet_manager.cc:1939] T 67e2461daa194fda98eb915a36f99209 P 6b00a6aadf6240b0b34a3c2594e2e0d7: Deleting consensus metadata
I20260806 02:56:21.221988 18740 ts_tablet_manager.cc:1918] T 67e2461daa194fda98eb915a36f99209 P ad3f7123cd8d4dcebd11197c8bfd5d39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:56:21.222221 18740 log.cc:1196] T 67e2461daa194fda98eb915a36f99209 P ad3f7123cd8d4dcebd11197c8bfd5d39: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-1/wal/wals/67e2461daa194fda98eb915a36f99209
I20260806 02:56:21.222990 18740 ts_tablet_manager.cc:1939] T 67e2461daa194fda98eb915a36f99209 P ad3f7123cd8d4dcebd11197c8bfd5d39: Deleting consensus metadata
I20260806 02:56:21.222913 17968 catalog_manager.cc:4841] TS 6b00a6aadf6240b0b34a3c2594e2e0d7 (127.25.211.131:43105): tablet 67e2461daa194fda98eb915a36f99209 (table test-table [id=805137b10abd4fc790fe63baa1290980]) successfully deleted
I20260806 02:56:21.225011 17973 catalog_manager.cc:4841] TS ad3f7123cd8d4dcebd11197c8bfd5d39 (127.25.211.130:42087): tablet 67e2461daa194fda98eb915a36f99209 (table test-table [id=805137b10abd4fc790fe63baa1290980]) successfully deleted
I20260806 02:56:21.312543 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 18042
W20260806 02:56:21.337981 17969 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.211.129:41493 (error 108)
I20260806 02:56:21.338502 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 18277
I20260806 02:56:21.340070 18695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:21.339646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:41493 (local address 127.0.0.1:45360)
0806 02:56:21.339827 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:21.339931 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.129:41493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260806 02:56:21.340307 17969 catalog_manager.cc:4584] TS e510bcdf18a54a4392a3392d53e84d4b (127.25.211.129:41493): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 67e2461daa194fda98eb915a36f99209: Network error: Client connection negotiation failed: client connection to 127.25.211.129:41493: connect: Connection refused (error 111)
I20260806 02:56:21.355854 18696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:21.339334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34560 (local address 127.25.211.130:42087)
0806 02:56:21.339542 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0806 02:56:21.339560 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 02:56:21.339632 (+ 72us) server_negotiation.cc:407] Connection header received
0806 02:56:21.340293 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:56:21.340300 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:56:21.340399 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:56:21.340509 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:56:21.342506 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:56:21.343276 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:56:21.345291 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:56:21.345757 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:56:21.345867 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:56:21.349415 (+ 3548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:56:21.349432 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:56:21.349450 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:56:21.349488 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:56:21.352857 (+ 3369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:56:21.353804 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:56:21.353812 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:56:21.353814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:56:21.353953 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:56:21.354396 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:56:21.354399 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:56:21.354401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:56:21.354800 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:56:21.355030 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0806 02:56:21.355374 (+ 344us) server_negotiation.cc:299] Negotiation successful
0806 02:56:21.355540 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 02:56:21.356675 18461 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:56:21.394176 18695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:21.393607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:41493 (local address 127.0.0.1:45372)
0806 02:56:21.393874 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:21.393967 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.129:41493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
I20260806 02:56:21.395421 18695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:21.394987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:41493 (local address 127.0.0.1:45388)
0806 02:56:21.395200 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:21.395293 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.129:41493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260806 02:56:21.456256 18695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:21.455685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:41493 (local address 127.0.0.1:45392)
0806 02:56:21.455928 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:21.456047 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.129:41493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260806 02:56:21.457473 18695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:21.457042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:41493 (local address 127.0.0.1:45402)
0806 02:56:21.457214 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:21.457314 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.129:41493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260806 02:56:21.462080 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 18277
I20260806 02:56:21.486943 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 18477
I20260806 02:56:21.503161 18697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:21.487826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38938 (local address 127.25.211.131:43105)
0806 02:56:21.488028 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0806 02:56:21.488034 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:56:21.488074 (+ 40us) server_negotiation.cc:407] Connection header received
0806 02:56:21.488205 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:56:21.488211 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:56:21.488300 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:56:21.488438 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:56:21.489711 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:56:21.490974 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:56:21.492342 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:56:21.493014 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:56:21.493170 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:56:21.495768 (+ 2598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:56:21.495792 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:56:21.495815 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:56:21.495869 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:56:21.500159 (+ 4290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:56:21.500860 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:56:21.500871 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:56:21.500875 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:56:21.501114 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:56:21.501472 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:56:21.501477 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:56:21.501481 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:56:21.501957 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:56:21.502331 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0806 02:56:21.502498 (+ 167us) server_negotiation.cc:299] Negotiation successful
0806 02:56:21.502740 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260806 02:56:21.503973 18675 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:56:21.538228 18695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:21.537669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:41493 (local address 127.0.0.1:45410)
0806 02:56:21.537903 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:21.538018 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.129:41493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260806 02:56:21.539288 18695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:21.538886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:41493 (local address 127.0.0.1:45422)
0806 02:56:21.539093 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:21.539172 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.129:41493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260806 02:56:21.627539 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 18477
I20260806 02:56:21.652931 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 17944
I20260806 02:56:21.654126 18032 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:56:21.769044 18695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:21.768388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:41493 (local address 127.0.0.1:45432)
0806 02:56:21.768638 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:21.768774 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.129:41493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260806 02:56:21.769199 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 17944
I20260806 02:56:41.794389 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:40273
--webserver_interface=127.25.211.190
--webserver_port=43535
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:37079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.211.190:40273
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:56:42.154656 18747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:56:42.154966 18747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:56:42.155102 18747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:56:42.162407 18747 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 02:56:42.162509 18747 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 02:56:42.162568 18747 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 02:56:42.162636 18747 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:56:42.162705 18747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:56:42.162806 18747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:56:42.162861 18747 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:56:42.162937 18747 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:56:42.173388 18747 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:37079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.25.211.190:40273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:40273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.211.190
--webserver_port=43535
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:56:42.174754 18747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:56:42.176100 18747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:56:42.188324 18753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:56:42.188421 18754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:56:42.188635 18747 server_base.cc:1034] running on GCE node
W20260806 02:56:42.188380 18757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:56:42.189704 18747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:56:42.191085 18747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:56:42.192310 18747 hybrid_clock.cc:648] HybridClock initialized: now 1785985002192248 us; error 29 us; skew 500 ppm
I20260806 02:56:42.192612 18747 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:56:42 dist-test-slave-zj8q krb5kdc[17141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985002, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:56:42.204986 18747 init.cc:377] Logged in from keytab as kudu/127.25.211.190@KRBTEST.COM (short username kudu)
I20260806 02:56:42.206756 18747 webserver.cc:460] Webserver started at http://127.25.211.190:43535/ using document root <none> and password file <none>
I20260806 02:56:42.207551 18747 fs_manager.cc:359] Metadata directory not provided
I20260806 02:56:42.207660 18747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:56:42.213665 18747 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260806 02:56:42.216756 18768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:56:42.218098 18747 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:56:42.218298 18747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/master-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/master-0/wal
uuid: "9fbba35f31c947459e6407cf737d30a9"
format_stamp: "Formatted at 2026-08-06 02:55:56 on dist-test-slave-zj8q"
server_key: "56ff814429ca0b8259b1b4798847d1ff"
server_key_iv: "f2e84ea6f9c0cc27182d3c90c9c0601f"
server_key_version: "encryptionkey@0"
I20260806 02:56:42.219285 18747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:56:42.234831 18747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:56:42.239297 18747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:56:42.239609 18747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:56:42.259052 18747 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.190:40273
I20260806 02:56:42.259083 18839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.190:40273 every 8 connection(s)
I20260806 02:56:42.260743 18747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
I20260806 02:56:42.266681 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 18747
I20260806 02:56:42.267599 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.129:41493
--local_ip_for_outbound_sockets=127.25.211.129
--tserver_master_addrs=127.25.211.190:40273
--webserver_port=33697
--webserver_interface=127.25.211.129
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab
--principal=kudu/127.25.211.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:37079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:56:42.268661 18840 sys_catalog.cc:263] Verifying existing consensus state
I20260806 02:56:42.272320 18840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9: Bootstrap starting.
I20260806 02:56:42.296929 18840 log.cc:824] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9: Log is configured to *not* fsync() on all Append() calls
I20260806 02:56:42.318607 18840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:56:42.319228 18840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9: Bootstrap complete.
I20260806 02:56:42.328845 18840 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fbba35f31c947459e6407cf737d30a9" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 40273 } }
I20260806 02:56:42.329787 18840 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fbba35f31c947459e6407cf737d30a9, State: Initialized, Role: FOLLOWER
I20260806 02:56:42.330444 18840 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fbba35f31c947459e6407cf737d30a9" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 40273 } }
I20260806 02:56:42.330688 18840 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:56:42.330809 18840 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:56:42.330974 18840 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9 [term 2 FOLLOWER]: Advancing to term 3
I20260806 02:56:42.335247 18840 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fbba35f31c947459e6407cf737d30a9" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 40273 } }
I20260806 02:56:42.335793 18840 leader_election.cc:302] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9fbba35f31c947459e6407cf737d30a9; no voters:
I20260806 02:56:42.336494 18840 leader_election.cc:288] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260806 02:56:42.336665 18844 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 02:56:42.338059 18844 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9 [term 3 LEADER]: Becoming Leader. State: Replica: 9fbba35f31c947459e6407cf737d30a9, State: Running, Role: LEADER
I20260806 02:56:42.338790 18844 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fbba35f31c947459e6407cf737d30a9" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 40273 } }
I20260806 02:56:42.339135 18840 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:56:42.341442 18847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fbba35f31c947459e6407cf737d30a9. Latest consensus state: current_term: 3 leader_uuid: "9fbba35f31c947459e6407cf737d30a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fbba35f31c947459e6407cf737d30a9" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 40273 } } }
I20260806 02:56:42.341867 18847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9 [sys.catalog]: This master's current role is: LEADER
I20260806 02:56:42.342371 18846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "9fbba35f31c947459e6407cf737d30a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fbba35f31c947459e6407cf737d30a9" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 40273 } } }
I20260806 02:56:42.342676 18846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9 [sys.catalog]: This master's current role is: LEADER
I20260806 02:56:42.347790 18853 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:56:42.357219 18853 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:56:42.358904 18853 catalog_manager.cc:1259] Loaded cluster ID: 918343e9b6c54837a2d7994365cdaa9d
I20260806 02:56:42.359084 18853 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:56:42.363330 18853 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:56:42.365172 18853 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9: Loaded TSK: 0
I20260806 02:56:42.365401 18853 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9fbba35f31c947459e6407cf737d30a9: Loaded TSK: 1
F20260806 02:56:42.366641 18853 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ff6a59b79ad google::LogMessage::Fail() at ??:0
@ 0x7ff6a59ba88a google::LogMessage::SendToLog() at ??:0
@ 0x7ff6a59b74de google::LogMessage::Flush() at ??:0
@ 0x7ff6a59b8a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff6a971b7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7ff6a970a73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7ff6bb3e61c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7ff6bb3e5783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7ff6bb4712a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7ff6bb4710c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7ff6bb4bb5e9 std::function<>::operator()() at ??:0
@ 0x7ff6bb3e9cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7ff6bb3e7c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ff6bb46f610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7ff6bb46f439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7ff6a8603345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff6a860691c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7ff6a8606799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7ff6a85e1eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff6ab8bb6db start_thread at ??:0
@ 0x7ff6a485161f clone at ??:0
W20260806 02:56:42.722182 18842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:56:42.722533 18842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:56:42.722684 18842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:56:42.730293 18842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:56:42.730441 18842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:56:42.730610 18842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.129
I20260806 02:56:42.740836 18842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:37079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab
--principal=kudu/127.25.211.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.129:41493
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.211.129
--webserver_port=33697
--tserver_master_addrs=127.25.211.190:40273
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.129
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:56:42.742247 18842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:56:42.743619 18842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:56:42.757509 18908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:56:42.757593 18911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:56:42.757596 18907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:56:42.758414 18842 server_base.cc:1034] running on GCE node
I20260806 02:56:42.758988 18842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:56:42.759948 18842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:56:42.761152 18842 hybrid_clock.cc:648] HybridClock initialized: now 1785985002761095 us; error 43 us; skew 500 ppm
I20260806 02:56:42.761467 18842 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:56:42 dist-test-slave-zj8q krb5kdc[17141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985002, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:56:42.764829 18842 init.cc:377] Logged in from keytab as kudu/127.25.211.129@KRBTEST.COM (short username kudu)
I20260806 02:56:42.766422 18842 webserver.cc:460] Webserver started at http://127.25.211.129:33697/ using document root <none> and password file <none>
I20260806 02:56:42.767132 18842 fs_manager.cc:359] Metadata directory not provided
I20260806 02:56:42.767263 18842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:56:42.773291 18842 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260806 02:56:42.776638 18920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:56:42.778120 18842 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:56:42.778359 18842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-0/wal
uuid: "e510bcdf18a54a4392a3392d53e84d4b"
format_stamp: "Formatted at 2026-08-06 02:55:56 on dist-test-slave-zj8q"
server_key: "94f068353ac3a82a0be9510dacbc6285"
server_key_iv: "528daddd51df4da6c848dbf8e452a33d"
server_key_version: "encryptionkey@0"
I20260806 02:56:42.779387 18842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:56:42.820350 18842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:56:42.824713 18842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:56:42.825011 18842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:56:42.826121 18842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:56:42.828166 18928 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260806 02:56:42.834215 18842 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 02:56:42.834357 18842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260806 02:56:42.834503 18842 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 02:56:42.837859 18842 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 02:56:42.837954 18842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20260806 02:56:42.838414 18928 tablet_bootstrap.cc:492] T 67e2461daa194fda98eb915a36f99209 P e510bcdf18a54a4392a3392d53e84d4b: Bootstrap starting.
I20260806 02:56:42.847548 18928 log.cc:824] T 67e2461daa194fda98eb915a36f99209 P e510bcdf18a54a4392a3392d53e84d4b: Log is configured to *not* fsync() on all Append() calls
I20260806 02:56:42.865912 18928 tablet_bootstrap.cc:492] T 67e2461daa194fda98eb915a36f99209 P e510bcdf18a54a4392a3392d53e84d4b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 02:56:42.866878 18928 tablet_bootstrap.cc:492] T 67e2461daa194fda98eb915a36f99209 P e510bcdf18a54a4392a3392d53e84d4b: Bootstrap complete.
I20260806 02:56:42.867619 18928 ts_tablet_manager.cc:1397] T 67e2461daa194fda98eb915a36f99209 P e510bcdf18a54a4392a3392d53e84d4b: Time spent bootstrapping tablet: real 0.030s user 0.023s sys 0.004s
I20260806 02:56:42.874436 18842 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.129:41493
I20260806 02:56:42.874476 19075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.129:41493 every 8 connection(s)
I20260806 02:56:42.876322 18842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
I20260806 02:56:42.877779 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 18842
I20260806 02:56:42.878654 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.130:42087
--local_ip_for_outbound_sockets=127.25.211.130
--tserver_master_addrs=127.25.211.190:40273
--webserver_port=34423
--webserver_interface=127.25.211.130
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab
--principal=kudu/127.25.211.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:37079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:56:42.879164 18928 raft_consensus.cc:348] T 67e2461daa194fda98eb915a36f99209 P e510bcdf18a54a4392a3392d53e84d4b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3f7123cd8d4dcebd11197c8bfd5d39" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 42087 } } peers { permanent_uuid: "6b00a6aadf6240b0b34a3c2594e2e0d7" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 43105 } } peers { permanent_uuid: "e510bcdf18a54a4392a3392d53e84d4b" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 41493 } }
I20260806 02:56:42.880730 18928 raft_consensus.cc:729] T 67e2461daa194fda98eb915a36f99209 P e510bcdf18a54a4392a3392d53e84d4b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e510bcdf18a54a4392a3392d53e84d4b, State: Initialized, Role: FOLLOWER
I20260806 02:56:42.881502 18928 consensus_queue.cc:260] T 67e2461daa194fda98eb915a36f99209 P e510bcdf18a54a4392a3392d53e84d4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3f7123cd8d4dcebd11197c8bfd5d39" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 42087 } } peers { permanent_uuid: "6b00a6aadf6240b0b34a3c2594e2e0d7" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 43105 } } peers { permanent_uuid: "e510bcdf18a54a4392a3392d53e84d4b" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 41493 } }
I20260806 02:56:42.888568 19078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:42.879857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:40273 (local address 127.25.211.129:42671)
0806 02:56:42.886880 (+ 7023us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:42.887251 (+ 371us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:40273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6625,"thread_start_us":6383,"threads_started":1}
I20260806 02:56:42.888675 18928 ts_tablet_manager.cc:1428] T 67e2461daa194fda98eb915a36f99209 P e510bcdf18a54a4392a3392d53e84d4b: Time spent starting tablet: real 0.021s user 0.012s sys 0.008s
W20260806 02:56:42.889278 19076 heartbeater.cc:644] Failed to heartbeat to 127.25.211.190:40273 (0 consecutive failures): Network error: Failed to ping master at 127.25.211.190:40273: Client connection negotiation failed: client connection to 127.25.211.190:40273: connect: Connection refused (error 111)
I20260806 02:56:42.890206 19078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:42.889853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:40273 (local address 127.25.211.129:45269)
0806 02:56:42.889991 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:42.890061 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:40273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260806 02:56:42.891008 19078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:42.890705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:40273 (local address 127.25.211.129:47649)
0806 02:56:42.890820 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:42.890894 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:40273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260806 02:56:42.891240 19076 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 02:56:43.230042 19080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:56:43.230394 19080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:56:43.230511 19080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:56:43.238000 19080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:56:43.238139 19080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:56:43.238312 19080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.130
I20260806 02:56:43.248744 19080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:37079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab
--principal=kudu/127.25.211.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.130:42087
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.211.130
--webserver_port=34423
--tserver_master_addrs=127.25.211.190:40273
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.130
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:56:43.250131 19080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:56:43.251657 19080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:56:43.265631 19093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:56:43.265659 19095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:56:43.265689 19100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:56:43.266642 19080 server_base.cc:1034] running on GCE node
I20260806 02:56:43.267282 19080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:56:43.268251 19080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:56:43.269490 19080 hybrid_clock.cc:648] HybridClock initialized: now 1785985003269427 us; error 50 us; skew 500 ppm
I20260806 02:56:43.269835 19080 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:56:43 dist-test-slave-zj8q krb5kdc[17141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985003, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:56:43.273283 19080 init.cc:377] Logged in from keytab as kudu/127.25.211.130@KRBTEST.COM (short username kudu)
I20260806 02:56:43.274852 19080 webserver.cc:460] Webserver started at http://127.25.211.130:34423/ using document root <none> and password file <none>
I20260806 02:56:43.275552 19080 fs_manager.cc:359] Metadata directory not provided
I20260806 02:56:43.275691 19080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:56:43.281438 19080 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:56:43.284559 19109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:56:43.286036 19080 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260806 02:56:43.286237 19080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-1/wal
uuid: "ad3f7123cd8d4dcebd11197c8bfd5d39"
format_stamp: "Formatted at 2026-08-06 02:55:57 on dist-test-slave-zj8q"
server_key: "a5d418862eaf3a46b7a9bac6cc8d1112"
server_key_iv: "165d0e598acfe0545342fadf5333ab47"
server_key_version: "encryptionkey@0"
I20260806 02:56:43.287256 19080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:56:43.310434 19080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:56:43.314790 19080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:56:43.315070 19080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:56:43.316118 19080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:56:43.317608 19080 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:56:43.317715 19080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:56:43.317968 19080 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:56:43.318082 19080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:56:43.350376 19080 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.130:42087
I20260806 02:56:43.350454 19264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.130:42087 every 8 connection(s)
I20260806 02:56:43.351944 19080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
I20260806 02:56:43.353518 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 19080
I20260806 02:56:43.354333 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.131:43105
--local_ip_for_outbound_sockets=127.25.211.131
--tserver_master_addrs=127.25.211.190:40273
--webserver_port=40627
--webserver_interface=127.25.211.131
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab
--principal=kudu/127.25.211.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:37079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:56:43.358058 19267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:43.355518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:40273 (local address 127.25.211.130:36733)
0806 02:56:43.356472 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:43.356800 (+ 328us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:40273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":345,"threads_started":1}
W20260806 02:56:43.358989 19265 heartbeater.cc:644] Failed to heartbeat to 127.25.211.190:40273 (0 consecutive failures): Network error: Failed to ping master at 127.25.211.190:40273: Client connection negotiation failed: client connection to 127.25.211.190:40273: connect: Connection refused (error 111)
I20260806 02:56:43.360034 19267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:43.359595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:40273 (local address 127.25.211.130:60295)
0806 02:56:43.359799 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:43.359890 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:40273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260806 02:56:43.361023 19267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:43.360651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:40273 (local address 127.25.211.130:37255)
0806 02:56:43.360798 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:43.360889 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:40273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260806 02:56:43.361274 19265 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 02:56:43.701407 19269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:56:43.701705 19269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:56:43.701828 19269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:56:43.709371 19269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:56:43.709501 19269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:56:43.709656 19269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.131
I20260806 02:56:43.719803 19269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:37079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab
--principal=kudu/127.25.211.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.131:43105
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.211.131
--webserver_port=40627
--tserver_master_addrs=127.25.211.190:40273
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.131
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:56:43.721196 19269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:56:43.722579 19269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:56:43.735699 19277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:56:43.735718 19276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:56:43.736315 19282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:56:43.736642 19269 server_base.cc:1034] running on GCE node
I20260806 02:56:43.737293 19269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:56:43.738358 19269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:56:43.739574 19269 hybrid_clock.cc:648] HybridClock initialized: now 1785985003739517 us; error 53 us; skew 500 ppm
I20260806 02:56:43.739919 19269 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:56:43 dist-test-slave-zj8q krb5kdc[17141](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985003, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:56:43.743064 19269 init.cc:377] Logged in from keytab as kudu/127.25.211.131@KRBTEST.COM (short username kudu)
I20260806 02:56:43.744524 19269 webserver.cc:460] Webserver started at http://127.25.211.131:40627/ using document root <none> and password file <none>
I20260806 02:56:43.745245 19269 fs_manager.cc:359] Metadata directory not provided
I20260806 02:56:43.745373 19269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:56:43.751078 19269 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 02:56:43.754091 19291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:56:43.755568 19269 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 02:56:43.755771 19269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-2/wal
uuid: "6b00a6aadf6240b0b34a3c2594e2e0d7"
format_stamp: "Formatted at 2026-08-06 02:55:57 on dist-test-slave-zj8q"
server_key: "a856e286e26af2352781b931775f42e6"
server_key_iv: "772930b073f49b4447b74462bb61c8d3"
server_key_version: "encryptionkey@0"
I20260806 02:56:43.756793 19269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:56:43.787954 19269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:56:43.792336 19269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:56:43.792623 19269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:56:43.793658 19269 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:56:43.795128 19269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:56:43.795236 19269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:56:43.795346 19269 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:56:43.795415 19269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:56:43.827972 19269 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.131:43105
I20260806 02:56:43.828070 19444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.131:43105 every 8 connection(s)
I20260806 02:56:43.829562 19269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
I20260806 02:56:43.834903 19450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:43.832771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:40273 (local address 127.25.211.131:57925)
0806 02:56:43.833590 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:43.833869 (+ 279us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:40273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":313,"threads_started":1}
W20260806 02:56:43.835737 19446 heartbeater.cc:644] Failed to heartbeat to 127.25.211.190:40273 (0 consecutive failures): Network error: Failed to ping master at 127.25.211.190:40273: Client connection negotiation failed: client connection to 127.25.211.190:40273: connect: Connection refused (error 111)
I20260806 02:56:43.836655 19450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:43.836296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:40273 (local address 127.25.211.131:54247)
0806 02:56:43.836456 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:43.836533 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:40273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260806 02:56:43.837330 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 19269
I20260806 02:56:43.837577 19450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:43.837223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:40273 (local address 127.25.211.131:53207)
0806 02:56:43.837366 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:43.837458 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:40273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260806 02:56:43.837823 19446 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 02:56:43.892768 19453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:43.891888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:40273 (local address 127.25.211.129:34431)
0806 02:56:43.892437 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:43.892539 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:40273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":268,"threads_started":1}
I20260806 02:56:44.167711 19454 raft_consensus.cc:482] T 67e2461daa194fda98eb915a36f99209 P e510bcdf18a54a4392a3392d53e84d4b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:56:44.168004 19454 raft_consensus.cc:504] T 67e2461daa194fda98eb915a36f99209 P e510bcdf18a54a4392a3392d53e84d4b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3f7123cd8d4dcebd11197c8bfd5d39" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 42087 } } peers { permanent_uuid: "6b00a6aadf6240b0b34a3c2594e2e0d7" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 43105 } } peers { permanent_uuid: "e510bcdf18a54a4392a3392d53e84d4b" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 41493 } }
I20260806 02:56:44.169673 19454 leader_election.cc:288] T 67e2461daa194fda98eb915a36f99209 P e510bcdf18a54a4392a3392d53e84d4b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad3f7123cd8d4dcebd11197c8bfd5d39 (127.25.211.130:42087), 6b00a6aadf6240b0b34a3c2594e2e0d7 (127.25.211.131:43105)
Aug 06 02:56:44 dist-test-slave-zj8q krb5kdc[17141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985002, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.129@KRBTEST.COM for kudu/127.25.211.131@KRBTEST.COM
Aug 06 02:56:44 dist-test-slave-zj8q krb5kdc[17141](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985002, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.129@KRBTEST.COM for kudu/127.25.211.130@KRBTEST.COM
I20260806 02:56:44.190935 19456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:44.170363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:55329 (local address 127.25.211.130:42087)
0806 02:56:44.170987 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0806 02:56:44.171003 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:56:44.171607 (+ 604us) server_negotiation.cc:407] Connection header received
0806 02:56:44.173108 (+ 1501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:56:44.173116 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:56:44.173444 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:56:44.173737 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:56:44.175601 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:56:44.176936 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:56:44.178967 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:56:44.179619 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:56:44.179817 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:56:44.184091 (+ 4274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:56:44.184099 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:56:44.184111 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:56:44.184158 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:56:44.187165 (+ 3007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:56:44.187972 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:56:44.187978 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:56:44.187984 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:56:44.188183 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:56:44.188616 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:56:44.188621 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:56:44.188624 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:56:44.189133 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:56:44.189332 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0806 02:56:44.190126 (+ 794us) server_negotiation.cc:299] Negotiation successful
0806 02:56:44.190508 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":273,"threads_started":1}
I20260806 02:56:44.191027 19453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:44.170053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.131:43105 (local address 127.25.211.129:47951)
0806 02:56:44.170296 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:44.170337 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 02:56:44.170946 (+ 609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:56:44.173736 (+ 2790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:56:44.173747 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:56:44.174649 (+ 902us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:56:44.175402 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:56:44.175423 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:56:44.177221 (+ 1798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:56:44.177227 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:56:44.178813 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:56:44.178828 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:56:44.179836 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:56:44.179843 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:56:44.180157 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:56:44.180896 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:56:44.180921 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:56:44.183502 (+ 2581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:56:44.187070 (+ 3568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:56:44.187079 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:56:44.187096 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:56:44.187634 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:56:44.188144 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:56:44.188149 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:56:44.188152 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:56:44.188386 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:56:44.189268 (+ 882us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:56:44.189276 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:56:44.189595 (+ 319us) client_negotiation.cc:769] Sending connection context
0806 02:56:44.189849 (+ 254us) client_negotiation.cc:240] Negotiation successful
0806 02:56:44.190584 (+ 735us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"mutex_wait_us":443,"spinlock_wait_cycles":20480}
I20260806 02:56:44.191035 19458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:44.170066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.130:42087 (local address 127.25.211.129:55329)
0806 02:56:44.171484 (+ 1418us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:44.171499 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:56:44.171626 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:56:44.173737 (+ 2111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:56:44.173750 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:56:44.174651 (+ 901us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:56:44.175401 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:56:44.175422 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:56:44.177149 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:56:44.177155 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:56:44.178817 (+ 1662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:56:44.178831 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:56:44.179886 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:56:44.179895 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:56:44.180155 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:56:44.180952 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:56:44.180970 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:56:44.183815 (+ 2845us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:56:44.187304 (+ 3489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:56:44.187309 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:56:44.187313 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:56:44.187838 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:56:44.188324 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:56:44.188328 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:56:44.188332 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:56:44.188491 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:56:44.189286 (+ 795us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:56:44.189291 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:56:44.189597 (+ 306us) client_negotiation.cc:769] Sending connection context
0806 02:56:44.189849 (+ 252us) client_negotiation.cc:240] Negotiation successful
0806 02:56:44.190454 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1297,"mutex_wait_us":806,"spinlock_wait_cycles":24576,"thread_start_us":465,"threads_started":1}
I20260806 02:56:44.191085 19457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:44.170417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:47951 (local address 127.25.211.131:43105)
0806 02:56:44.171352 (+ 935us) server_negotiation.cc:206] Beginning negotiation
0806 02:56:44.171362 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:56:44.171397 (+ 35us) server_negotiation.cc:407] Connection header received
0806 02:56:44.172868 (+ 1471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:56:44.172876 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:56:44.173245 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:56:44.173563 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:56:44.175605 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:56:44.177036 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:56:44.178960 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:56:44.179620 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:56:44.179805 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:56:44.183842 (+ 4037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:56:44.183850 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:56:44.183866 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:56:44.183904 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:56:44.186917 (+ 3013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:56:44.187787 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:56:44.187794 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:56:44.187799 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:56:44.188019 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:56:44.188519 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:56:44.188524 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:56:44.188526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:56:44.189121 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:56:44.189313 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0806 02:56:44.190127 (+ 814us) server_negotiation.cc:299] Negotiation successful
0806 02:56:44.190537 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":721,"thread_start_us":547,"threads_started":1}
W20260806 02:56:44.194778 18924 leader_election.cc:341] T 67e2461daa194fda98eb915a36f99209 P e510bcdf18a54a4392a3392d53e84d4b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad3f7123cd8d4dcebd11197c8bfd5d39 (127.25.211.130:42087): Not found: Tablet not found: 67e2461daa194fda98eb915a36f99209
W20260806 02:56:44.195086 18921 leader_election.cc:341] T 67e2461daa194fda98eb915a36f99209 P e510bcdf18a54a4392a3392d53e84d4b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6b00a6aadf6240b0b34a3c2594e2e0d7 (127.25.211.131:43105): Not found: Tablet not found: 67e2461daa194fda98eb915a36f99209
I20260806 02:56:44.195220 18921 leader_election.cc:302] T 67e2461daa194fda98eb915a36f99209 P e510bcdf18a54a4392a3392d53e84d4b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e510bcdf18a54a4392a3392d53e84d4b; no voters: 6b00a6aadf6240b0b34a3c2594e2e0d7, ad3f7123cd8d4dcebd11197c8bfd5d39
I20260806 02:56:44.195600 19454 raft_consensus.cc:2726] T 67e2461daa194fda98eb915a36f99209 P e510bcdf18a54a4392a3392d53e84d4b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 02:56:44.362841 19459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:44.361859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:40273 (local address 127.25.211.130:48495)
0806 02:56:44.362398 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:44.362563 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:40273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":237,"threads_started":1}
I20260806 02:56:44.839298 19460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:44.838492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:40273 (local address 127.25.211.131:50515)
0806 02:56:44.838955 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:44.839080 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:40273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":218,"threads_started":1}
I20260806 02:56:44.894565 19461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:44.893788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:40273 (local address 127.25.211.129:47681)
0806 02:56:44.894284 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:44.894383 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:40273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":218,"threads_started":1}
I20260806 02:56:45.364755 19462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:45.363920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:40273 (local address 127.25.211.130:56679)
0806 02:56:45.364419 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:45.364506 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:40273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":260,"threads_started":1}
I20260806 02:56:45.840958 19463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:45.840205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:40273 (local address 127.25.211.131:35699)
0806 02:56:45.840662 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:45.840768 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:40273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":239,"threads_started":1}
I20260806 02:56:45.896421 19465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:45.895556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:40273 (local address 127.25.211.129:47105)
0806 02:56:45.896100 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:45.896212 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:40273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":233,"threads_started":1}
I20260806 02:56:46.068428 19466 raft_consensus.cc:482] T 67e2461daa194fda98eb915a36f99209 P e510bcdf18a54a4392a3392d53e84d4b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:56:46.068638 19466 raft_consensus.cc:504] T 67e2461daa194fda98eb915a36f99209 P e510bcdf18a54a4392a3392d53e84d4b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3f7123cd8d4dcebd11197c8bfd5d39" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 42087 } } peers { permanent_uuid: "6b00a6aadf6240b0b34a3c2594e2e0d7" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 43105 } } peers { permanent_uuid: "e510bcdf18a54a4392a3392d53e84d4b" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 41493 } }
I20260806 02:56:46.069319 19466 leader_election.cc:288] T 67e2461daa194fda98eb915a36f99209 P e510bcdf18a54a4392a3392d53e84d4b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad3f7123cd8d4dcebd11197c8bfd5d39 (127.25.211.130:42087), 6b00a6aadf6240b0b34a3c2594e2e0d7 (127.25.211.131:43105)
W20260806 02:56:46.070369 18921 leader_election.cc:341] T 67e2461daa194fda98eb915a36f99209 P e510bcdf18a54a4392a3392d53e84d4b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6b00a6aadf6240b0b34a3c2594e2e0d7 (127.25.211.131:43105): Not found: Tablet not found: 67e2461daa194fda98eb915a36f99209
W20260806 02:56:46.070513 18924 leader_election.cc:341] T 67e2461daa194fda98eb915a36f99209 P e510bcdf18a54a4392a3392d53e84d4b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad3f7123cd8d4dcebd11197c8bfd5d39 (127.25.211.130:42087): Not found: Tablet not found: 67e2461daa194fda98eb915a36f99209
I20260806 02:56:46.070710 18924 leader_election.cc:302] T 67e2461daa194fda98eb915a36f99209 P e510bcdf18a54a4392a3392d53e84d4b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e510bcdf18a54a4392a3392d53e84d4b; no voters: 6b00a6aadf6240b0b34a3c2594e2e0d7, ad3f7123cd8d4dcebd11197c8bfd5d39
I20260806 02:56:46.070976 19466 raft_consensus.cc:2726] T 67e2461daa194fda98eb915a36f99209 P e510bcdf18a54a4392a3392d53e84d4b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 02:56:46.366581 19468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:46.365744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:40273 (local address 127.25.211.130:58497)
0806 02:56:46.366302 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:46.366395 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:40273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":251,"threads_started":1}
I20260806 02:56:46.842746 19469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:46.841949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:40273 (local address 127.25.211.131:49495)
0806 02:56:46.842463 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:46.842563 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:40273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":284,"threads_started":1}
I20260806 02:56:46.898180 19471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:46.897375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:40273 (local address 127.25.211.129:43387)
0806 02:56:46.897893 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:46.897984 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:40273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":294,"threads_started":1}
I20260806 02:56:47.368125 19472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:47.367428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:40273 (local address 127.25.211.130:32883)
0806 02:56:47.367855 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:47.367939 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:40273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":200,"threads_started":1}
I20260806 02:56:47.844373 19473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:47.843611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:40273 (local address 127.25.211.131:33237)
0806 02:56:47.844086 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:47.844178 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:40273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":241,"threads_started":1}
I20260806 02:56:47.900081 19474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:47.899244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:40273 (local address 127.25.211.129:55423)
0806 02:56:47.899779 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:47.899880 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:40273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":270,"threads_started":1}
I20260806 02:56:48.369834 19475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:48.369050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:40273 (local address 127.25.211.130:42159)
0806 02:56:48.369530 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:48.369637 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:40273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":268,"threads_started":1}
I20260806 02:56:48.375316 19477 raft_consensus.cc:482] T 67e2461daa194fda98eb915a36f99209 P e510bcdf18a54a4392a3392d53e84d4b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:56:48.375521 19477 raft_consensus.cc:504] T 67e2461daa194fda98eb915a36f99209 P e510bcdf18a54a4392a3392d53e84d4b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3f7123cd8d4dcebd11197c8bfd5d39" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 42087 } } peers { permanent_uuid: "6b00a6aadf6240b0b34a3c2594e2e0d7" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 43105 } } peers { permanent_uuid: "e510bcdf18a54a4392a3392d53e84d4b" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 41493 } }
I20260806 02:56:48.376199 19477 leader_election.cc:288] T 67e2461daa194fda98eb915a36f99209 P e510bcdf18a54a4392a3392d53e84d4b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad3f7123cd8d4dcebd11197c8bfd5d39 (127.25.211.130:42087), 6b00a6aadf6240b0b34a3c2594e2e0d7 (127.25.211.131:43105)
W20260806 02:56:48.377244 18921 leader_election.cc:341] T 67e2461daa194fda98eb915a36f99209 P e510bcdf18a54a4392a3392d53e84d4b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6b00a6aadf6240b0b34a3c2594e2e0d7 (127.25.211.131:43105): Not found: Tablet not found: 67e2461daa194fda98eb915a36f99209
W20260806 02:56:48.377441 18924 leader_election.cc:341] T 67e2461daa194fda98eb915a36f99209 P e510bcdf18a54a4392a3392d53e84d4b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad3f7123cd8d4dcebd11197c8bfd5d39 (127.25.211.130:42087): Not found: Tablet not found: 67e2461daa194fda98eb915a36f99209
I20260806 02:56:48.377625 18924 leader_election.cc:302] T 67e2461daa194fda98eb915a36f99209 P e510bcdf18a54a4392a3392d53e84d4b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e510bcdf18a54a4392a3392d53e84d4b; no voters: 6b00a6aadf6240b0b34a3c2594e2e0d7, ad3f7123cd8d4dcebd11197c8bfd5d39
I20260806 02:56:48.377863 19477 raft_consensus.cc:2726] T 67e2461daa194fda98eb915a36f99209 P e510bcdf18a54a4392a3392d53e84d4b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 02:56:48.846124 19480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:48.845314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:40273 (local address 127.25.211.131:47559)
0806 02:56:48.845867 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:48.845956 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:40273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":310,"threads_started":1}
I20260806 02:56:48.901808 19481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:48.901056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:40273 (local address 127.25.211.129:53441)
0806 02:56:48.901509 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:48.901627 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:40273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":242,"threads_started":1}
I20260806 02:56:49.371572 19483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:49.370781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:40273 (local address 127.25.211.130:48177)
0806 02:56:49.371303 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:49.371390 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:40273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":272,"threads_started":1}
I20260806 02:56:49.847951 19485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:49.847158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:40273 (local address 127.25.211.131:50833)
0806 02:56:49.847673 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:49.847766 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:40273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":251,"threads_started":1}
I20260806 02:56:49.903640 19486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:49.902799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:40273 (local address 127.25.211.129:56953)
0806 02:56:49.903305 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:49.903415 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:40273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":251,"threads_started":1}
I20260806 02:56:50.373365 19487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:50.372560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:40273 (local address 127.25.211.130:40299)
0806 02:56:50.373075 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:50.373175 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:40273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":220,"threads_started":1}
I20260806 02:56:50.850032 19488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:50.848978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:40273 (local address 127.25.211.131:54433)
0806 02:56:50.849719 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:50.849817 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:40273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":266,"threads_started":1}
I20260806 02:56:50.905503 19489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:50.904727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:40273 (local address 127.25.211.129:36169)
0806 02:56:50.905229 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:50.905334 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:40273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":246,"threads_started":1}
I20260806 02:56:51.375298 19491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:51.374418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:40273 (local address 127.25.211.130:34949)
0806 02:56:51.374950 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:51.375068 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:40273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":289,"threads_started":1}
I20260806 02:56:51.851794 19492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:51.851022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:40273 (local address 127.25.211.131:48113)
0806 02:56:51.851480 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:51.851594 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:40273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":240,"threads_started":1}
I20260806 02:56:51.907383 19494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:51.906502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:40273 (local address 127.25.211.129:42135)
0806 02:56:51.907101 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:51.907189 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:40273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":362,"threads_started":1}
I20260806 02:56:52.377148 19495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:52.376267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:40273 (local address 127.25.211.130:55517)
0806 02:56:52.376878 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:52.376983 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:40273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":294,"threads_started":1}
I20260806 02:56:52.581581 19496 raft_consensus.cc:482] T 67e2461daa194fda98eb915a36f99209 P e510bcdf18a54a4392a3392d53e84d4b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:56:52.581784 19496 raft_consensus.cc:504] T 67e2461daa194fda98eb915a36f99209 P e510bcdf18a54a4392a3392d53e84d4b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad3f7123cd8d4dcebd11197c8bfd5d39" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 42087 } } peers { permanent_uuid: "6b00a6aadf6240b0b34a3c2594e2e0d7" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 43105 } } peers { permanent_uuid: "e510bcdf18a54a4392a3392d53e84d4b" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 41493 } }
I20260806 02:56:52.582576 19496 leader_election.cc:288] T 67e2461daa194fda98eb915a36f99209 P e510bcdf18a54a4392a3392d53e84d4b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ad3f7123cd8d4dcebd11197c8bfd5d39 (127.25.211.130:42087), 6b00a6aadf6240b0b34a3c2594e2e0d7 (127.25.211.131:43105)
W20260806 02:56:52.583600 18924 leader_election.cc:341] T 67e2461daa194fda98eb915a36f99209 P e510bcdf18a54a4392a3392d53e84d4b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad3f7123cd8d4dcebd11197c8bfd5d39 (127.25.211.130:42087): Not found: Tablet not found: 67e2461daa194fda98eb915a36f99209
W20260806 02:56:52.583796 18921 leader_election.cc:341] T 67e2461daa194fda98eb915a36f99209 P e510bcdf18a54a4392a3392d53e84d4b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6b00a6aadf6240b0b34a3c2594e2e0d7 (127.25.211.131:43105): Not found: Tablet not found: 67e2461daa194fda98eb915a36f99209
I20260806 02:56:52.583954 18921 leader_election.cc:302] T 67e2461daa194fda98eb915a36f99209 P e510bcdf18a54a4392a3392d53e84d4b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e510bcdf18a54a4392a3392d53e84d4b; no voters: 6b00a6aadf6240b0b34a3c2594e2e0d7, ad3f7123cd8d4dcebd11197c8bfd5d39
I20260806 02:56:52.584249 19496 raft_consensus.cc:2726] T 67e2461daa194fda98eb915a36f99209 P e510bcdf18a54a4392a3392d53e84d4b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 02:56:52.853441 19497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:52.852691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:40273 (local address 127.25.211.131:58927)
0806 02:56:52.853196 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:52.853285 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:40273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":238,"threads_started":1}
I20260806 02:56:52.909329 19499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:52.908443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:40273 (local address 127.25.211.129:60131)
0806 02:56:52.908988 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:52.909122 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:40273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":281,"threads_started":1}
I20260806 02:56:53.379051 19500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:53.378187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:40273 (local address 127.25.211.130:55019)
0806 02:56:53.378741 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:53.378839 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:40273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":283,"threads_started":1}
I20260806 02:56:53.855262 19501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:53.854540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:40273 (local address 127.25.211.131:33133)
0806 02:56:53.854986 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:53.855074 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:40273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":225,"threads_started":1}
I20260806 02:56:53.911187 19504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:53.910435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:40273 (local address 127.25.211.129:46793)
0806 02:56:53.910904 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:53.910998 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:40273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":244,"threads_started":1}
I20260806 02:56:54.380793 19505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:54.380040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:40273 (local address 127.25.211.130:46265)
0806 02:56:54.380528 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:54.380630 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:40273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":238,"threads_started":1}
I20260806 02:56:54.857076 19507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:54.856255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:40273 (local address 127.25.211.131:38487)
0806 02:56:54.856800 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:54.856904 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:40273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":266,"threads_started":1}
I20260806 02:56:54.912983 19508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:54.912166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:40273 (local address 127.25.211.129:54323)
0806 02:56:54.912675 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:54.912802 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:40273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":281,"threads_started":1}
I20260806 02:56:55.382674 19509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:55.381856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:40273 (local address 127.25.211.130:37297)
0806 02:56:55.382384 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:55.382487 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:40273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":294,"threads_started":1}
I20260806 02:56:55.858942 19510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:55.858089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:40273 (local address 127.25.211.131:49433)
0806 02:56:55.858681 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:55.858768 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:40273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":287,"threads_started":1}
I20260806 02:56:55.914935 19511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:55.914094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:40273 (local address 127.25.211.129:41625)
0806 02:56:55.914651 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:55.914750 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:40273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":317,"threads_started":1}
I20260806 02:56:56.384624 19513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:56.383834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:40273 (local address 127.25.211.130:44939)
0806 02:56:56.384332 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:56.384437 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:40273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":243,"threads_started":1}
I20260806 02:56:56.861347 19514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:56.860098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:40273 (local address 127.25.211.131:49955)
0806 02:56:56.861059 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:56.861158 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:40273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":301,"threads_started":1}
I20260806 02:56:56.916790 19515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:56.915898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:40273 (local address 127.25.211.129:59919)
0806 02:56:56.916459 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:56.916550 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:40273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":295,"threads_started":1}
I20260806 02:56:57.386489 19516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:57.385585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:40273 (local address 127.25.211.130:50627)
0806 02:56:57.386133 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:57.386274 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:40273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":269,"threads_started":1}
I20260806 02:56:57.863303 19517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:57.862383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:40273 (local address 127.25.211.131:60019)
0806 02:56:57.862946 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:57.863050 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:40273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":337,"threads_started":1}
I20260806 02:56:57.919005 19520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:57.918073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:40273 (local address 127.25.211.129:45383)
0806 02:56:57.918701 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:57.918800 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:40273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":306,"threads_started":1}
I20260806 02:56:58.388470 19523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:58.387675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:40273 (local address 127.25.211.130:54643)
0806 02:56:58.388180 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:58.388296 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:40273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":273,"threads_started":1}
I20260806 02:56:58.844576 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 18842
I20260806 02:56:58.860136 19524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:58.845764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53176 (local address 127.25.211.129:41493)
0806 02:56:58.846401 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0806 02:56:58.846410 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:56:58.846440 (+ 30us) server_negotiation.cc:407] Connection header received
0806 02:56:58.846504 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:56:58.846510 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:56:58.846729 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:56:58.846862 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:56:58.848272 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:56:58.849252 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:56:58.850706 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:56:58.851347 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:56:58.851466 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:56:58.854254 (+ 2788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:56:58.854270 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:56:58.854287 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:56:58.854327 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:56:58.857213 (+ 2886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:56:58.858018 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:56:58.858026 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:56:58.858028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:56:58.858257 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:56:58.858641 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:56:58.858645 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:56:58.858647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:56:58.859041 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:56:58.859303 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0806 02:56:58.859550 (+ 247us) server_negotiation.cc:299] Negotiation successful
0806 02:56:58.859783 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":327,"threads_started":1}
I20260806 02:56:58.861570 19069 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:56:58.865610 19526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:58.864668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:40273 (local address 127.25.211.131:45141)
0806 02:56:58.865252 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0806 02:56:58.865397 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.190:40273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":264,"threads_started":1}
I20260806 02:56:58.952049 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 18842
I20260806 02:56:58.972925 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 19080
I20260806 02:56:58.987108 19528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:58.973926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46238 (local address 127.25.211.130:42087)
0806 02:56:58.974432 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0806 02:56:58.974437 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:56:58.974455 (+ 18us) server_negotiation.cc:407] Connection header received
0806 02:56:58.974511 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:56:58.974515 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:56:58.974576 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:56:58.974667 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:56:58.976074 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:56:58.976849 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:56:58.978120 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:56:58.978777 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:56:58.978888 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:56:58.981631 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:56:58.981636 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:56:58.981640 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:56:58.981676 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:56:58.984211 (+ 2535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:56:58.984973 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:56:58.984977 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:56:58.984979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:56:58.985131 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:56:58.985532 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:56:58.985536 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:56:58.985538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:56:58.985811 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:56:58.986017 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0806 02:56:58.986518 (+ 501us) server_negotiation.cc:299] Negotiation successful
0806 02:56:58.986724 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":291,"threads_started":1}
I20260806 02:56:58.987841 19258 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:56:59.072728 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 19080
I20260806 02:56:59.096297 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 19269
I20260806 02:56:59.110968 19532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:56:59.097352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59528 (local address 127.25.211.131:43105)
0806 02:56:59.097938 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0806 02:56:59.097943 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:56:59.097961 (+ 18us) server_negotiation.cc:407] Connection header received
0806 02:56:59.098019 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:56:59.098023 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:56:59.098102 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:56:59.098229 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:56:59.099685 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:56:59.100555 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:56:59.101834 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:56:59.102392 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:56:59.102508 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:56:59.105162 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:56:59.105168 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:56:59.105172 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:56:59.105208 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:56:59.108057 (+ 2849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:56:59.108818 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:56:59.108821 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:56:59.108824 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:56:59.108973 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:56:59.109334 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:56:59.109337 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:56:59.109339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:56:59.109636 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:56:59.109876 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0806 02:56:59.110434 (+ 558us) server_negotiation.cc:299] Negotiation successful
0806 02:56:59.110625 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":321,"threads_started":1}
I20260806 02:56:59.111682 19440 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:56:59.194931 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 19269
2026-08-06T02:56:59Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (65058 ms)
[----------] 30 tests from SecurityITest (312031 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:56:59 dist-test-slave-zj8q krb5kdc[19536](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:56:59 dist-test-slave-zj8q krb5kdc[19536](info): set up 2 sockets
Aug 06 02:56:59 dist-test-slave-zj8q krb5kdc[19536](info): commencing operation
krb5kdc: starting...
W20260806 02:57:00.432991 26446 mini_kdc.cc:121] Time spent starting KDC: real 1.191s user 0.002s sys 0.046s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Aug 06 02:57:00 dist-test-slave-zj8q krb5kdc[19536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:57:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:57:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.211.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab.
Entry for principal kudu/127.25.211.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab.
I20260806 02:57:00.759003 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:44459
--webserver_interface=127.25.211.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:37637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.211.190:44459
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:57:01.118331 19555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:57:01.118678 19555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:57:01.118826 19555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:57:01.126430 19555 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:57:01.126574 19555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:57:01.126654 19555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:57:01.126737 19555 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:57:01.126801 19555 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:57:01.137071 19555 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:37637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.211.190:44459
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:44459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.211.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:57:01.138572 19555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:57:01.139897 19555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:57:01.152135 19565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:01.152177 19566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:01.152199 19568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:57:01.152863 19555 server_base.cc:1034] running on GCE node
I20260806 02:57:01.153688 19555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:57:01.155203 19555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:57:01.156432 19555 hybrid_clock.cc:648] HybridClock initialized: now 1785985021156388 us; error 28 us; skew 500 ppm
I20260806 02:57:01.156793 19555 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:57:01 dist-test-slave-zj8q krb5kdc[19536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985021, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:57:01.165443 19555 init.cc:377] Logged in from keytab as kudu/127.25.211.190@KRBTEST.COM (short username kudu)
I20260806 02:57:01.167109 19555 webserver.cc:460] Webserver started at http://127.25.211.190:40455/ using document root <none> and password file <none>
I20260806 02:57:01.167840 19555 fs_manager.cc:359] Metadata directory not provided
I20260806 02:57:01.167965 19555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:57:01.168290 19555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:57:01.171566 19555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/master-0/data/instance:
uuid: "b8b5a4bebd144b038472d34987e3088c"
format_stamp: "Formatted at 2026-08-06 02:57:01 on dist-test-slave-zj8q"
server_key: "b518949831efe5883df0e1b79ccd28de"
server_key_iv: "912361ac2e40b9d523cefc31c79415dd"
server_key_version: "encryptionkey@0"
I20260806 02:57:01.172366 19555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/master-0/wal/instance:
uuid: "b8b5a4bebd144b038472d34987e3088c"
format_stamp: "Formatted at 2026-08-06 02:57:01 on dist-test-slave-zj8q"
server_key: "b518949831efe5883df0e1b79ccd28de"
server_key_iv: "912361ac2e40b9d523cefc31c79415dd"
server_key_version: "encryptionkey@0"
I20260806 02:57:01.177826 19555 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260806 02:57:01.181526 19576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:01.182909 19555 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260806 02:57:01.183095 19555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/master-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/master-0/wal
uuid: "b8b5a4bebd144b038472d34987e3088c"
format_stamp: "Formatted at 2026-08-06 02:57:01 on dist-test-slave-zj8q"
server_key: "b518949831efe5883df0e1b79ccd28de"
server_key_iv: "912361ac2e40b9d523cefc31c79415dd"
server_key_version: "encryptionkey@0"
I20260806 02:57:01.183319 19555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:57:01.197180 19555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:57:01.201377 19555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:57:01.201655 19555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:57:01.219972 19555 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.190:44459
I20260806 02:57:01.220006 19650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.190:44459 every 8 connection(s)
I20260806 02:57:01.221446 19555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
I20260806 02:57:01.225494 19651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:57:01.231398 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 19555
I20260806 02:57:01.231607 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/master-0/wal/instance
I20260806 02:57:01.232266 26446 external_mini_cluster.cc:1428] Setting key 9f32beb21bc5cfa217dacb9db6e702f4
I20260806 02:57:01.237883 19651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8b5a4bebd144b038472d34987e3088c: Bootstrap starting.
I20260806 02:57:01.241918 19651 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8b5a4bebd144b038472d34987e3088c: Neither blocks nor log segments found. Creating new log.
I20260806 02:57:01.243151 19651 log.cc:824] T 00000000000000000000000000000000 P b8b5a4bebd144b038472d34987e3088c: Log is configured to *not* fsync() on all Append() calls
I20260806 02:57:01.246459 19651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8b5a4bebd144b038472d34987e3088c: No bootstrap required, opened a new log
Aug 06 02:57:01 dist-test-slave-zj8q krb5kdc[19536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:57:01.256438 19651 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8b5a4bebd144b038472d34987e3088c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8b5a4bebd144b038472d34987e3088c" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 44459 } }
I20260806 02:57:01.256973 19651 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8b5a4bebd144b038472d34987e3088c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:57:01.257089 19651 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8b5a4bebd144b038472d34987e3088c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8b5a4bebd144b038472d34987e3088c, State: Initialized, Role: FOLLOWER
I20260806 02:57:01.257799 19651 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8b5a4bebd144b038472d34987e3088c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8b5a4bebd144b038472d34987e3088c" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 44459 } }
I20260806 02:57:01.258088 19651 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8b5a4bebd144b038472d34987e3088c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:57:01.258272 19651 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8b5a4bebd144b038472d34987e3088c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:57:01.258477 19651 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8b5a4bebd144b038472d34987e3088c [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:57:01.258522 19655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:01.234443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53978 (local address 127.25.211.190:44459)
0806 02:57:01.235307 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:01.235324 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:01.235378 (+ 54us) server_negotiation.cc:407] Connection header received
0806 02:57:01.237757 (+ 2379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:01.237790 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:01.238376 (+ 586us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:01.238856 (+ 480us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:01.240247 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:01.242320 (+ 2073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:01.243830 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:01.244587 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:01.244771 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:01.249206 (+ 4435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:01.249222 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:01.249245 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:57:01.249295 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:01.253548 (+ 4253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:01.254393 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:01.254404 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:01.254414 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:01.254650 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:01.255071 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:01.255076 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:01.255079 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:01.255887 (+ 808us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:01.256184 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:01.256578 (+ 394us) server_negotiation.cc:299] Negotiation successful
0806 02:57:01.257059 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":350,"threads_started":1}
I20260806 02:57:01.260960 19651 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8b5a4bebd144b038472d34987e3088c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8b5a4bebd144b038472d34987e3088c" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 44459 } }
I20260806 02:57:01.261452 19651 leader_election.cc:302] T 00000000000000000000000000000000 P b8b5a4bebd144b038472d34987e3088c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8b5a4bebd144b038472d34987e3088c; no voters:
I20260806 02:57:01.262041 19651 leader_election.cc:288] T 00000000000000000000000000000000 P b8b5a4bebd144b038472d34987e3088c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:57:01.262205 19657 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8b5a4bebd144b038472d34987e3088c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:57:01.262894 19657 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8b5a4bebd144b038472d34987e3088c [term 1 LEADER]: Becoming Leader. State: Replica: b8b5a4bebd144b038472d34987e3088c, State: Running, Role: LEADER
I20260806 02:57:01.263466 19657 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8b5a4bebd144b038472d34987e3088c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8b5a4bebd144b038472d34987e3088c" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 44459 } }
I20260806 02:57:01.264050 19651 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8b5a4bebd144b038472d34987e3088c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:57:01.266937 19661 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8b5a4bebd144b038472d34987e3088c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8b5a4bebd144b038472d34987e3088c. Latest consensus state: current_term: 1 leader_uuid: "b8b5a4bebd144b038472d34987e3088c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8b5a4bebd144b038472d34987e3088c" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 44459 } } }
I20260806 02:57:01.267366 19661 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8b5a4bebd144b038472d34987e3088c [sys.catalog]: This master's current role is: LEADER
I20260806 02:57:01.268929 19670 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:57:01.269045 19660 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8b5a4bebd144b038472d34987e3088c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8b5a4bebd144b038472d34987e3088c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8b5a4bebd144b038472d34987e3088c" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 44459 } } }
I20260806 02:57:01.269383 19660 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8b5a4bebd144b038472d34987e3088c [sys.catalog]: This master's current role is: LEADER
I20260806 02:57:01.274613 19670 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:57:01.284626 19670 catalog_manager.cc:1347] Generated new cluster ID: 6baeac8258224c8c9e0772810350eecf
I20260806 02:57:01.284755 19670 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:57:01.301918 19670 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:57:01.303431 19670 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:57:01.311064 19670 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8b5a4bebd144b038472d34987e3088c: Generated new TSK 0
I20260806 02:57:01.311957 19670 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.211.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab.
Entry for principal kudu/127.25.211.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab.
I20260806 02:57:01.395927 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.129:0
--local_ip_for_outbound_sockets=127.25.211.129
--webserver_interface=127.25.211.129
--webserver_port=0
--tserver_master_addrs=127.25.211.190:44459
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab
--principal=kudu/127.25.211.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:37637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:57:01.788684 19686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:57:01.788975 19686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:57:01.789083 19686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:57:01.796310 19686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:57:01.796435 19686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:57:01.796550 19686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.129
I20260806 02:57:01.806480 19686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:37637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab
--principal=kudu/127.25.211.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.211.129
--webserver_port=0
--tserver_master_addrs=127.25.211.190:44459
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.129
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:57:01.807852 19686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:57:01.809231 19686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:57:01.822893 19695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:01.822930 19697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:01.822999 19692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:57:01.823829 19686 server_base.cc:1034] running on GCE node
I20260806 02:57:01.824538 19686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:57:01.825678 19686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:57:01.826918 19686 hybrid_clock.cc:648] HybridClock initialized: now 1785985021826853 us; error 53 us; skew 500 ppm
I20260806 02:57:01.827272 19686 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:57:01 dist-test-slave-zj8q krb5kdc[19536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985021, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:57:01.830596 19686 init.cc:377] Logged in from keytab as kudu/127.25.211.129@KRBTEST.COM (short username kudu)
I20260806 02:57:01.832141 19686 webserver.cc:460] Webserver started at http://127.25.211.129:32873/ using document root <none> and password file <none>
I20260806 02:57:01.832830 19686 fs_manager.cc:359] Metadata directory not provided
I20260806 02:57:01.832933 19686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:57:01.833202 19686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:57:01.836107 19686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/ts-0/data/instance:
uuid: "55a15f23e9ce445089da79455f1ff981"
format_stamp: "Formatted at 2026-08-06 02:57:01 on dist-test-slave-zj8q"
server_key: "dc8f5eb5e6fa2e24cc304186ca782a6b"
server_key_iv: "5b7bab2495f6f8f68680f6c2c7b80db1"
server_key_version: "encryptionkey@0"
I20260806 02:57:01.836824 19686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/ts-0/wal/instance:
uuid: "55a15f23e9ce445089da79455f1ff981"
format_stamp: "Formatted at 2026-08-06 02:57:01 on dist-test-slave-zj8q"
server_key: "dc8f5eb5e6fa2e24cc304186ca782a6b"
server_key_iv: "5b7bab2495f6f8f68680f6c2c7b80db1"
server_key_version: "encryptionkey@0"
I20260806 02:57:01.842037 19686 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260806 02:57:01.845444 19706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:01.846830 19686 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:57:01.846976 19686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/ts-0/wal
uuid: "55a15f23e9ce445089da79455f1ff981"
format_stamp: "Formatted at 2026-08-06 02:57:01 on dist-test-slave-zj8q"
server_key: "dc8f5eb5e6fa2e24cc304186ca782a6b"
server_key_iv: "5b7bab2495f6f8f68680f6c2c7b80db1"
server_key_version: "encryptionkey@0"
I20260806 02:57:01.847155 19686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:57:01.879514 19686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:57:01.883714 19686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:57:01.883970 19686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:57:01.884967 19686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:57:01.886401 19686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:57:01.886497 19686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:01.886605 19686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:57:01.886647 19686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:01.918107 19686 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.129:34237
I20260806 02:57:01.918205 19867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.129:34237 every 8 connection(s)
I20260806 02:57:01.919478 19686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
I20260806 02:57:01.923882 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 19686
I20260806 02:57:01.924120 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/ts-0/wal/instance
I20260806 02:57:01.925040 26446 external_mini_cluster.cc:1428] Setting key f6a5749fccd0040ee61a6bace0520041
Aug 06 02:57:01 dist-test-slave-zj8q krb5kdc[19536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985021, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.129@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:57:01.941464 19870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:01.921938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:48467 (local address 127.25.211.190:44459)
0806 02:57:01.922460 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:01.922467 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:01.923383 (+ 916us) server_negotiation.cc:407] Connection header received
0806 02:57:01.925938 (+ 2555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:01.925944 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:01.926025 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:01.926126 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:01.928072 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:01.928823 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:01.930999 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:01.931419 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:01.931528 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:01.935507 (+ 3979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:01.935513 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:01.935516 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:57:01.935574 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:01.938051 (+ 2477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:01.938945 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:01.938950 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:01.938952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:01.939092 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:01.939605 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:01.939609 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:01.939611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:01.939909 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:01.940029 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:01.940843 (+ 814us) server_negotiation.cc:299] Negotiation successful
0806 02:57:01.941095 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":262,"threads_started":1}
I20260806 02:57:01.942193 19871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:01.922508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:44459 (local address 127.25.211.129:48467)
0806 02:57:01.923205 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0806 02:57:01.923256 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 02:57:01.925612 (+ 2356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:57:01.926412 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:57:01.926420 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:57:01.927055 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:57:01.927873 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:01.927900 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:01.928995 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:01.929001 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:01.930831 (+ 1830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:01.930847 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:01.931651 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:01.931657 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:01.931917 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:01.932602 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:57:01.932630 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:57:01.935256 (+ 2626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:57:01.938211 (+ 2955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:57:01.938220 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:57:01.938242 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:57:01.938816 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:57:01.939190 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:57:01.939194 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:57:01.939197 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:57:01.939426 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:57:01.940030 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:57:01.940037 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:57:01.940441 (+ 404us) client_negotiation.cc:769] Sending connection context
0806 02:57:01.940668 (+ 227us) client_negotiation.cc:240] Negotiation successful
0806 02:57:01.941061 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":273,"threads_started":1}
I20260806 02:57:01.943707 19868 heartbeater.cc:347] Connected to a master server at 127.25.211.190:44459
I20260806 02:57:01.944061 19868 heartbeater.cc:464] Registering TS with master...
I20260806 02:57:01.944926 19868 heartbeater.cc:511] Master 127.25.211.190:44459 requested a full tablet report, sending...
I20260806 02:57:01.947330 19600 ts_manager.cc:194] Registered new tserver with Master: 55a15f23e9ce445089da79455f1ff981 (127.25.211.129:34237)
I20260806 02:57:01.948840 19600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.211.129@KRBTEST.COM'} at 127.25.211.129:48467
WARNING: no policy specified for kudu/127.25.211.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab.
Entry for principal kudu/127.25.211.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab.
I20260806 02:57:01.996178 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.130:0
--local_ip_for_outbound_sockets=127.25.211.130
--webserver_interface=127.25.211.130
--webserver_port=0
--tserver_master_addrs=127.25.211.190:44459
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab
--principal=kudu/127.25.211.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:37637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:57:02.346602 19874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:57:02.346932 19874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:57:02.347072 19874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:57:02.354655 19874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:57:02.354794 19874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:57:02.354956 19874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.130
I20260806 02:57:02.365123 19874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:37637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab
--principal=kudu/127.25.211.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.211.130
--webserver_port=0
--tserver_master_addrs=127.25.211.190:44459
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.130
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:57:02.366544 19874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:57:02.367967 19874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:57:02.381595 19879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:02.381632 19883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:02.381655 19880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:57:02.382375 19874 server_base.cc:1034] running on GCE node
I20260806 02:57:02.383055 19874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:57:02.384078 19874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:57:02.385316 19874 hybrid_clock.cc:648] HybridClock initialized: now 1785985022385261 us; error 42 us; skew 500 ppm
I20260806 02:57:02.385703 19874 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:57:02 dist-test-slave-zj8q krb5kdc[19536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985022, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:57:02.389251 19874 init.cc:377] Logged in from keytab as kudu/127.25.211.130@KRBTEST.COM (short username kudu)
I20260806 02:57:02.390893 19874 webserver.cc:460] Webserver started at http://127.25.211.130:45383/ using document root <none> and password file <none>
I20260806 02:57:02.391575 19874 fs_manager.cc:359] Metadata directory not provided
I20260806 02:57:02.391681 19874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:57:02.391993 19874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:57:02.395151 19874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/ts-1/data/instance:
uuid: "48bd2ee03a684fbfa40afa51a907faa7"
format_stamp: "Formatted at 2026-08-06 02:57:02 on dist-test-slave-zj8q"
server_key: "17d5633d1bbdc29795219f9432b0335d"
server_key_iv: "cffea22d7241e8aa6c92ee63e7e08315"
server_key_version: "encryptionkey@0"
I20260806 02:57:02.395978 19874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/ts-1/wal/instance:
uuid: "48bd2ee03a684fbfa40afa51a907faa7"
format_stamp: "Formatted at 2026-08-06 02:57:02 on dist-test-slave-zj8q"
server_key: "17d5633d1bbdc29795219f9432b0335d"
server_key_iv: "cffea22d7241e8aa6c92ee63e7e08315"
server_key_version: "encryptionkey@0"
I20260806 02:57:02.401681 19874 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260806 02:57:02.405496 19893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:02.406968 19874 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:57:02.407150 19874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/ts-1/wal
uuid: "48bd2ee03a684fbfa40afa51a907faa7"
format_stamp: "Formatted at 2026-08-06 02:57:02 on dist-test-slave-zj8q"
server_key: "17d5633d1bbdc29795219f9432b0335d"
server_key_iv: "cffea22d7241e8aa6c92ee63e7e08315"
server_key_version: "encryptionkey@0"
I20260806 02:57:02.407351 19874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:57:02.434940 19874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:57:02.439215 19874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:57:02.439496 19874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:57:02.440500 19874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:57:02.441963 19874 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:57:02.442063 19874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:02.442207 19874 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:57:02.442272 19874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260806 02:57:02.473551 19874 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.130:35823
I20260806 02:57:02.473595 20057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.130:35823 every 8 connection(s)
I20260806 02:57:02.474922 19874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
I20260806 02:57:02.480461 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 19874
I20260806 02:57:02.480639 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/ts-1/wal/instance
I20260806 02:57:02.481108 26446 external_mini_cluster.cc:1428] Setting key 3dff49173197e8bdbf0bb5be189a1977
Aug 06 02:57:02 dist-test-slave-zj8q krb5kdc[19536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985022, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.130@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:57:02.494859 20066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:02.477522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:48609 (local address 127.25.211.190:44459)
0806 02:57:02.477957 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:02.477962 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:02.479020 (+ 1058us) server_negotiation.cc:407] Connection header received
0806 02:57:02.480906 (+ 1886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:02.480911 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:02.480997 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:02.481092 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:02.482809 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:02.483631 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:02.485211 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:02.485674 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:02.485794 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:02.489306 (+ 3512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:02.489311 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:02.489314 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:57:02.489347 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:02.491838 (+ 2491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:02.492543 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:02.492547 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:02.492549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:02.492693 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:02.493112 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:02.493116 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:02.493118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:02.493385 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:02.493517 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:02.494247 (+ 730us) server_negotiation.cc:299] Negotiation successful
0806 02:57:02.494477 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":208,"threads_started":1}
I20260806 02:57:02.495896 20067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:02.478052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:44459 (local address 127.25.211.130:48609)
0806 02:57:02.478849 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0806 02:57:02.478896 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 02:57:02.480610 (+ 1714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:57:02.481347 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:57:02.481356 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:57:02.482008 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:57:02.482640 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:02.482658 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:02.483791 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:02.483795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:02.485066 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:02.485079 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:02.485905 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:02.485912 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:02.486193 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:02.486866 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:57:02.486901 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:57:02.489077 (+ 2176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:57:02.491978 (+ 2901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:57:02.491988 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:57:02.492004 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:57:02.492422 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:57:02.492799 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:57:02.492804 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:57:02.492808 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:57:02.492997 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:57:02.493507 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:57:02.493514 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:57:02.493851 (+ 337us) client_negotiation.cc:769] Sending connection context
0806 02:57:02.494082 (+ 231us) client_negotiation.cc:240] Negotiation successful
0806 02:57:02.494636 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":285,"threads_started":1}
I20260806 02:57:02.497359 20064 heartbeater.cc:347] Connected to a master server at 127.25.211.190:44459
I20260806 02:57:02.497694 20064 heartbeater.cc:464] Registering TS with master...
I20260806 02:57:02.498560 20064 heartbeater.cc:511] Master 127.25.211.190:44459 requested a full tablet report, sending...
I20260806 02:57:02.500154 19600 ts_manager.cc:194] Registered new tserver with Master: 48bd2ee03a684fbfa40afa51a907faa7 (127.25.211.130:35823)
I20260806 02:57:02.500957 19600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.211.130@KRBTEST.COM'} at 127.25.211.130:48609
WARNING: no policy specified for kudu/127.25.211.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab.
Entry for principal kudu/127.25.211.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab.
I20260806 02:57:02.549824 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.131:0
--local_ip_for_outbound_sockets=127.25.211.131
--webserver_interface=127.25.211.131
--webserver_port=0
--tserver_master_addrs=127.25.211.190:44459
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab
--principal=kudu/127.25.211.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:37637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:57:02.897060 20070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:57:02.897341 20070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:57:02.897437 20070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:57:02.904893 20070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:57:02.905028 20070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:57:02.905144 20070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.131
I20260806 02:57:02.915441 20070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:37637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab
--principal=kudu/127.25.211.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.211.131
--webserver_port=0
--tserver_master_addrs=127.25.211.190:44459
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.131
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:57:02.916703 20070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:57:02.918188 20070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:57:02.931258 20076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:02.931296 20075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:02.931346 20079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:57:02.932147 20070 server_base.cc:1034] running on GCE node
I20260806 02:57:02.932775 20070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:57:02.933728 20070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:57:02.934932 20070 hybrid_clock.cc:648] HybridClock initialized: now 1785985022934864 us; error 60 us; skew 500 ppm
I20260806 02:57:02.935228 20070 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:57:02 dist-test-slave-zj8q krb5kdc[19536](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985022, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:57:02.938655 20070 init.cc:377] Logged in from keytab as kudu/127.25.211.131@KRBTEST.COM (short username kudu)
I20260806 02:57:02.940171 20070 webserver.cc:460] Webserver started at http://127.25.211.131:38901/ using document root <none> and password file <none>
I20260806 02:57:02.940909 20070 fs_manager.cc:359] Metadata directory not provided
I20260806 02:57:02.941042 20070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:57:02.941365 20070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:57:02.944430 20070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/ts-2/data/instance:
uuid: "ad9b499e0c844724bc038bdc0cc37f74"
format_stamp: "Formatted at 2026-08-06 02:57:02 on dist-test-slave-zj8q"
server_key: "c24a3bd5b189a6482cdb7d07dbf75b30"
server_key_iv: "b57cb52cfd8a79fc320bb7cc9dd88785"
server_key_version: "encryptionkey@0"
I20260806 02:57:02.945209 20070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/ts-2/wal/instance:
uuid: "ad9b499e0c844724bc038bdc0cc37f74"
format_stamp: "Formatted at 2026-08-06 02:57:02 on dist-test-slave-zj8q"
server_key: "c24a3bd5b189a6482cdb7d07dbf75b30"
server_key_iv: "b57cb52cfd8a79fc320bb7cc9dd88785"
server_key_version: "encryptionkey@0"
I20260806 02:57:02.950822 20070 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260806 02:57:02.952302 19868 heartbeater.cc:503] Master 127.25.211.190:44459 was elected leader, sending a full tablet report...
I20260806 02:57:02.954862 20086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:02.956193 20070 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:57:02.956341 20070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/ts-2/wal
uuid: "ad9b499e0c844724bc038bdc0cc37f74"
format_stamp: "Formatted at 2026-08-06 02:57:02 on dist-test-slave-zj8q"
server_key: "c24a3bd5b189a6482cdb7d07dbf75b30"
server_key_iv: "b57cb52cfd8a79fc320bb7cc9dd88785"
server_key_version: "encryptionkey@0"
I20260806 02:57:02.956534 20070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:57:02.981954 20070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:57:02.986641 20070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:57:02.986938 20070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:57:02.988091 20070 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:57:02.989492 20070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:57:02.989567 20070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:02.989674 20070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:57:02.989725 20070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:03.021056 20070 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.131:46053
I20260806 02:57:03.021103 20247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.131:46053 every 8 connection(s)
I20260806 02:57:03.022456 20070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
I20260806 02:57:03.022938 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 20070
I20260806 02:57:03.023109 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/ts-2/wal/instance
I20260806 02:57:03.023682 26446 external_mini_cluster.cc:1428] Setting key e86011ff9ba38c6206f1572df1dd711a
Aug 06 02:57:03 dist-test-slave-zj8q krb5kdc[19536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985022, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.131@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:57:03.043345 20253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:03.025067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:41643 (local address 127.25.211.190:44459)
0806 02:57:03.025554 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:03.025560 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:03.026823 (+ 1263us) server_negotiation.cc:407] Connection header received
0806 02:57:03.028671 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:03.028676 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:03.028744 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:03.028839 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:03.030466 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:03.031278 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:03.032899 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:03.033321 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:03.033425 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:03.036968 (+ 3543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:03.036973 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:03.036976 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:57:03.037011 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:03.040294 (+ 3283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:03.041046 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:03.041050 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:03.041052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:03.041175 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:03.041598 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:03.041604 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:03.041606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:03.041934 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:03.042089 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:03.042844 (+ 755us) server_negotiation.cc:299] Negotiation successful
0806 02:57:03.043033 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":240,"threads_started":1}
I20260806 02:57:03.044296 20255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:03.025686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:44459 (local address 127.25.211.131:41643)
0806 02:57:03.026640 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0806 02:57:03.026701 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0806 02:57:03.028385 (+ 1684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:57:03.029102 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:57:03.029111 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:57:03.029656 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:57:03.030284 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:03.030306 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:03.031429 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:03.031433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:03.032720 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:03.032735 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:03.033517 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:03.033521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:03.033770 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:03.034484 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:57:03.034508 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:57:03.036741 (+ 2233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:57:03.040433 (+ 3692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:57:03.040441 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:57:03.040457 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:57:03.040947 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:57:03.041253 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:57:03.041256 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:57:03.041258 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:57:03.041439 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:57:03.042087 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:57:03.042093 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:57:03.042484 (+ 391us) client_negotiation.cc:769] Sending connection context
0806 02:57:03.042731 (+ 247us) client_negotiation.cc:240] Negotiation successful
0806 02:57:03.043155 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"thread_start_us":395,"threads_started":1}
I20260806 02:57:03.045889 20248 heartbeater.cc:347] Connected to a master server at 127.25.211.190:44459
I20260806 02:57:03.046221 20248 heartbeater.cc:464] Registering TS with master...
I20260806 02:57:03.046957 20248 heartbeater.cc:511] Master 127.25.211.190:44459 requested a full tablet report, sending...
I20260806 02:57:03.048456 19600 ts_manager.cc:194] Registered new tserver with Master: ad9b499e0c844724bc038bdc0cc37f74 (127.25.211.131:46053)
I20260806 02:57:03.049286 19600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.211.131@KRBTEST.COM'} at 127.25.211.131:41643
I20260806 02:57:03.052289 26446 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:57:03.069653 20253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:03.056914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35700 (local address 127.25.211.190:44459)
0806 02:57:03.057131 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:03.057139 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:03.057455 (+ 316us) server_negotiation.cc:407] Connection header received
0806 02:57:03.057603 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:03.057607 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:03.057696 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:03.057777 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:03.059324 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:03.060123 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:03.061515 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:03.061950 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:03.062046 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:03.064506 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:03.064511 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:03.064515 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:57:03.064579 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:03.067150 (+ 2571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:03.067913 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:03.067916 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:03.067919 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:03.068053 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:03.068343 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:03.068345 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:03.068347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:03.068691 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:03.068869 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:03.069154 (+ 285us) server_negotiation.cc:299] Negotiation successful
0806 02:57:03.069331 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 02:57:03.073940 19600 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35700:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:57:03.081707 19600 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:57:03.116849 20271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:03.094803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.130:35823 (local address 127.0.0.1:38238)
0806 02:57:03.105995 (+ 11192us) negotiation.cc:107] Waiting for socket to connect
0806 02:57:03.106048 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 02:57:03.107107 (+ 1059us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:57:03.108313 (+ 1206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:57:03.108320 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:57:03.108355 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:57:03.108847 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:03.108865 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:03.111324 (+ 2459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:03.111330 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:03.113991 (+ 2661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:03.114011 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:03.116101 (+ 2090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:03.116108 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:03.116262 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:03.116301 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 02:57:03.116385 (+ 84us) client_negotiation.cc:240] Negotiation successful
0806 02:57:03.116522 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10904,"thread_start_us":12916,"threads_started":1}
I20260806 02:57:03.118184 20270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:03.094985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:34237 (local address 127.0.0.1:34244)
0806 02:57:03.107992 (+ 13007us) negotiation.cc:107] Waiting for socket to connect
0806 02:57:03.108011 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:57:03.108226 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:57:03.109250 (+ 1024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:57:03.109256 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:57:03.109296 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:57:03.109739 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:03.109750 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:03.113525 (+ 3775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:03.113531 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:03.115545 (+ 2014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:03.115557 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:03.117392 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:03.117400 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:03.117567 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:03.117598 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 02:57:03.117665 (+ 67us) client_negotiation.cc:240] Negotiation successful
0806 02:57:03.117807 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12882,"thread_start_us":10559,"threads_started":1}
I20260806 02:57:03.118391 20272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:03.094992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38238 (local address 127.25.211.130:35823)
0806 02:57:03.107712 (+ 12720us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:03.107720 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:03.107751 (+ 31us) server_negotiation.cc:407] Connection header received
0806 02:57:03.107831 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:03.107837 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:03.108046 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:03.109090 (+ 1044us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:57:03.109379 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:03.111150 (+ 1771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:03.114276 (+ 3126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:03.115881 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:03.117185 (+ 1304us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:03.117781 (+ 596us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:03.117853 (+ 72us) server_negotiation.cc:299] Negotiation successful
0806 02:57:03.118041 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12577,"thread_start_us":12455,"threads_started":1}
I20260806 02:57:03.119467 20273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:03.095662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34244 (local address 127.25.211.129:34237)
0806 02:57:03.106895 (+ 11233us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:03.106907 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:03.108107 (+ 1200us) server_negotiation.cc:407] Connection header received
0806 02:57:03.108463 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:03.108473 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:03.108732 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:03.108906 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:57:03.109950 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:03.113312 (+ 3362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:03.115708 (+ 2396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:03.117152 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:03.118769 (+ 1617us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:03.118939 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:03.119004 (+ 65us) server_negotiation.cc:299] Negotiation successful
0806 02:57:03.119162 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11017,"thread_start_us":11633,"threads_started":1}
I20260806 02:57:03.122344 19964 tablet_service.cc:1467] Processing CreateTablet for tablet a21d22db7c2047049fa1dbd0688ffa6c (DEFAULT_TABLE table=test-table [id=723f1d77b106444c9c6a03568eb16314]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:57:03.122979 19775 tablet_service.cc:1467] Processing CreateTablet for tablet a21d22db7c2047049fa1dbd0688ffa6c (DEFAULT_TABLE table=test-table [id=723f1d77b106444c9c6a03568eb16314]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:57:03.123960 19964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a21d22db7c2047049fa1dbd0688ffa6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:57:03.124238 19775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a21d22db7c2047049fa1dbd0688ffa6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:57:03.124965 20275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:03.108546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33516 (local address 127.25.211.131:46053)
0806 02:57:03.116165 (+ 7619us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:03.116174 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:03.116208 (+ 34us) server_negotiation.cc:407] Connection header received
0806 02:57:03.116317 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:03.116325 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:03.116593 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:03.116809 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:57:03.117385 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:03.119899 (+ 2514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:03.122210 (+ 2311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:03.123594 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:03.123788 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:03.123953 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:03.124471 (+ 518us) server_negotiation.cc:299] Negotiation successful
0806 02:57:03.124644 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7435,"thread_start_us":7282,"threads_started":1}
I20260806 02:57:03.124591 20274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:03.108169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.131:46053 (local address 127.0.0.1:33516)
0806 02:57:03.114855 (+ 6686us) negotiation.cc:107] Waiting for socket to connect
0806 02:57:03.114875 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 02:57:03.114985 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:57:03.116818 (+ 1833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:57:03.116823 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:57:03.116855 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:57:03.117173 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:03.117183 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:03.120044 (+ 2861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:03.120049 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:03.121987 (+ 1938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:03.122004 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:03.123805 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:03.123812 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:03.123978 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:03.124013 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 02:57:03.124113 (+ 100us) client_negotiation.cc:240] Negotiation successful
0806 02:57:03.124253 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6542,"thread_start_us":6888,"threads_started":1}
I20260806 02:57:03.129570 20153 tablet_service.cc:1467] Processing CreateTablet for tablet a21d22db7c2047049fa1dbd0688ffa6c (DEFAULT_TABLE table=test-table [id=723f1d77b106444c9c6a03568eb16314]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:57:03.131141 20153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a21d22db7c2047049fa1dbd0688ffa6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:57:03.135771 20276 tablet_bootstrap.cc:492] T a21d22db7c2047049fa1dbd0688ffa6c P 48bd2ee03a684fbfa40afa51a907faa7: Bootstrap starting.
I20260806 02:57:03.138998 20276 tablet_bootstrap.cc:654] T a21d22db7c2047049fa1dbd0688ffa6c P 48bd2ee03a684fbfa40afa51a907faa7: Neither blocks nor log segments found. Creating new log.
I20260806 02:57:03.139864 20277 tablet_bootstrap.cc:492] T a21d22db7c2047049fa1dbd0688ffa6c P 55a15f23e9ce445089da79455f1ff981: Bootstrap starting.
I20260806 02:57:03.140381 20276 log.cc:824] T a21d22db7c2047049fa1dbd0688ffa6c P 48bd2ee03a684fbfa40afa51a907faa7: Log is configured to *not* fsync() on all Append() calls
I20260806 02:57:03.143476 20277 tablet_bootstrap.cc:654] T a21d22db7c2047049fa1dbd0688ffa6c P 55a15f23e9ce445089da79455f1ff981: Neither blocks nor log segments found. Creating new log.
I20260806 02:57:03.143734 20276 tablet_bootstrap.cc:492] T a21d22db7c2047049fa1dbd0688ffa6c P 48bd2ee03a684fbfa40afa51a907faa7: No bootstrap required, opened a new log
I20260806 02:57:03.144027 20276 ts_tablet_manager.cc:1397] T a21d22db7c2047049fa1dbd0688ffa6c P 48bd2ee03a684fbfa40afa51a907faa7: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260806 02:57:03.144810 20277 log.cc:824] T a21d22db7c2047049fa1dbd0688ffa6c P 55a15f23e9ce445089da79455f1ff981: Log is configured to *not* fsync() on all Append() calls
I20260806 02:57:03.151386 20279 tablet_bootstrap.cc:492] T a21d22db7c2047049fa1dbd0688ffa6c P ad9b499e0c844724bc038bdc0cc37f74: Bootstrap starting.
I20260806 02:57:03.154453 20277 tablet_bootstrap.cc:492] T a21d22db7c2047049fa1dbd0688ffa6c P 55a15f23e9ce445089da79455f1ff981: No bootstrap required, opened a new log
I20260806 02:57:03.154791 20277 ts_tablet_manager.cc:1397] T a21d22db7c2047049fa1dbd0688ffa6c P 55a15f23e9ce445089da79455f1ff981: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.004s
I20260806 02:57:03.155683 20276 raft_consensus.cc:348] T a21d22db7c2047049fa1dbd0688ffa6c P 48bd2ee03a684fbfa40afa51a907faa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48bd2ee03a684fbfa40afa51a907faa7" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 35823 } } peers { permanent_uuid: "55a15f23e9ce445089da79455f1ff981" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 34237 } } peers { permanent_uuid: "ad9b499e0c844724bc038bdc0cc37f74" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 46053 } }
I20260806 02:57:03.156064 20279 tablet_bootstrap.cc:654] T a21d22db7c2047049fa1dbd0688ffa6c P ad9b499e0c844724bc038bdc0cc37f74: Neither blocks nor log segments found. Creating new log.
I20260806 02:57:03.156211 20276 raft_consensus.cc:374] T a21d22db7c2047049fa1dbd0688ffa6c P 48bd2ee03a684fbfa40afa51a907faa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:57:03.156323 20276 raft_consensus.cc:729] T a21d22db7c2047049fa1dbd0688ffa6c P 48bd2ee03a684fbfa40afa51a907faa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48bd2ee03a684fbfa40afa51a907faa7, State: Initialized, Role: FOLLOWER
I20260806 02:57:03.157009 20276 consensus_queue.cc:260] T a21d22db7c2047049fa1dbd0688ffa6c P 48bd2ee03a684fbfa40afa51a907faa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48bd2ee03a684fbfa40afa51a907faa7" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 35823 } } peers { permanent_uuid: "55a15f23e9ce445089da79455f1ff981" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 34237 } } peers { permanent_uuid: "ad9b499e0c844724bc038bdc0cc37f74" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 46053 } }
I20260806 02:57:03.157423 20279 log.cc:824] T a21d22db7c2047049fa1dbd0688ffa6c P ad9b499e0c844724bc038bdc0cc37f74: Log is configured to *not* fsync() on all Append() calls
I20260806 02:57:03.162266 20064 heartbeater.cc:503] Master 127.25.211.190:44459 was elected leader, sending a full tablet report...
I20260806 02:57:03.163074 20276 ts_tablet_manager.cc:1428] T a21d22db7c2047049fa1dbd0688ffa6c P 48bd2ee03a684fbfa40afa51a907faa7: Time spent starting tablet: real 0.019s user 0.011s sys 0.009s
I20260806 02:57:03.164294 20279 tablet_bootstrap.cc:492] T a21d22db7c2047049fa1dbd0688ffa6c P ad9b499e0c844724bc038bdc0cc37f74: No bootstrap required, opened a new log
I20260806 02:57:03.163966 20277 raft_consensus.cc:348] T a21d22db7c2047049fa1dbd0688ffa6c P 55a15f23e9ce445089da79455f1ff981 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48bd2ee03a684fbfa40afa51a907faa7" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 35823 } } peers { permanent_uuid: "55a15f23e9ce445089da79455f1ff981" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 34237 } } peers { permanent_uuid: "ad9b499e0c844724bc038bdc0cc37f74" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 46053 } }
I20260806 02:57:03.164469 20277 raft_consensus.cc:374] T a21d22db7c2047049fa1dbd0688ffa6c P 55a15f23e9ce445089da79455f1ff981 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:57:03.164583 20277 raft_consensus.cc:729] T a21d22db7c2047049fa1dbd0688ffa6c P 55a15f23e9ce445089da79455f1ff981 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55a15f23e9ce445089da79455f1ff981, State: Initialized, Role: FOLLOWER
I20260806 02:57:03.164734 20279 ts_tablet_manager.cc:1397] T a21d22db7c2047049fa1dbd0688ffa6c P ad9b499e0c844724bc038bdc0cc37f74: Time spent bootstrapping tablet: real 0.014s user 0.012s sys 0.000s
I20260806 02:57:03.165271 20277 consensus_queue.cc:260] T a21d22db7c2047049fa1dbd0688ffa6c P 55a15f23e9ce445089da79455f1ff981 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48bd2ee03a684fbfa40afa51a907faa7" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 35823 } } peers { permanent_uuid: "55a15f23e9ce445089da79455f1ff981" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 34237 } } peers { permanent_uuid: "ad9b499e0c844724bc038bdc0cc37f74" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 46053 } }
I20260806 02:57:03.167222 20277 ts_tablet_manager.cc:1428] T a21d22db7c2047049fa1dbd0688ffa6c P 55a15f23e9ce445089da79455f1ff981: Time spent starting tablet: real 0.012s user 0.012s sys 0.001s
W20260806 02:57:03.171742 19869 tablet.cc:2369] T a21d22db7c2047049fa1dbd0688ffa6c P 55a15f23e9ce445089da79455f1ff981: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:57:03.173430 20279 raft_consensus.cc:348] T a21d22db7c2047049fa1dbd0688ffa6c P ad9b499e0c844724bc038bdc0cc37f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48bd2ee03a684fbfa40afa51a907faa7" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 35823 } } peers { permanent_uuid: "55a15f23e9ce445089da79455f1ff981" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 34237 } } peers { permanent_uuid: "ad9b499e0c844724bc038bdc0cc37f74" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 46053 } }
I20260806 02:57:03.173779 20279 raft_consensus.cc:374] T a21d22db7c2047049fa1dbd0688ffa6c P ad9b499e0c844724bc038bdc0cc37f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:57:03.173879 20279 raft_consensus.cc:729] T a21d22db7c2047049fa1dbd0688ffa6c P ad9b499e0c844724bc038bdc0cc37f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad9b499e0c844724bc038bdc0cc37f74, State: Initialized, Role: FOLLOWER
I20260806 02:57:03.174512 20279 consensus_queue.cc:260] T a21d22db7c2047049fa1dbd0688ffa6c P ad9b499e0c844724bc038bdc0cc37f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48bd2ee03a684fbfa40afa51a907faa7" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 35823 } } peers { permanent_uuid: "55a15f23e9ce445089da79455f1ff981" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 34237 } } peers { permanent_uuid: "ad9b499e0c844724bc038bdc0cc37f74" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 46053 } }
I20260806 02:57:03.175559 20248 heartbeater.cc:503] Master 127.25.211.190:44459 was elected leader, sending a full tablet report...
I20260806 02:57:03.176054 20279 ts_tablet_manager.cc:1428] T a21d22db7c2047049fa1dbd0688ffa6c P ad9b499e0c844724bc038bdc0cc37f74: Time spent starting tablet: real 0.011s user 0.006s sys 0.005s
W20260806 02:57:03.226958 20065 tablet.cc:2369] T a21d22db7c2047049fa1dbd0688ffa6c P 48bd2ee03a684fbfa40afa51a907faa7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:57:03.252552 20282 raft_consensus.cc:482] T a21d22db7c2047049fa1dbd0688ffa6c P 48bd2ee03a684fbfa40afa51a907faa7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:57:03.252873 20282 raft_consensus.cc:504] T a21d22db7c2047049fa1dbd0688ffa6c P 48bd2ee03a684fbfa40afa51a907faa7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48bd2ee03a684fbfa40afa51a907faa7" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 35823 } } peers { permanent_uuid: "55a15f23e9ce445089da79455f1ff981" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 34237 } } peers { permanent_uuid: "ad9b499e0c844724bc038bdc0cc37f74" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 46053 } }
I20260806 02:57:03.254635 20282 leader_election.cc:288] T a21d22db7c2047049fa1dbd0688ffa6c P 48bd2ee03a684fbfa40afa51a907faa7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 55a15f23e9ce445089da79455f1ff981 (127.25.211.129:34237), ad9b499e0c844724bc038bdc0cc37f74 (127.25.211.131:46053)
I20260806 02:57:03.262388 20275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:03.255265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:59399 (local address 127.25.211.131:46053)
0806 02:57:03.255463 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:03.255469 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:03.255844 (+ 375us) server_negotiation.cc:407] Connection header received
0806 02:57:03.256021 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:03.256047 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:03.256139 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:03.256243 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:57:03.256883 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:03.257948 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:03.260555 (+ 2607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:03.261368 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:03.261496 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:03.261563 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:03.261963 (+ 400us) server_negotiation.cc:299] Negotiation successful
0806 02:57:03.262106 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260806 02:57:03.262409 20273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:03.255225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:55763 (local address 127.25.211.129:34237)
0806 02:57:03.255437 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:03.255443 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:03.256562 (+ 1119us) server_negotiation.cc:407] Connection header received
0806 02:57:03.256708 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:03.256712 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:03.256784 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:03.256873 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:57:03.257326 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:03.258318 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:03.260654 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:03.261459 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:03.261623 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:03.261696 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:03.262014 (+ 318us) server_negotiation.cc:299] Negotiation successful
0806 02:57:03.262143 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 02:57:03.262533 20287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:03.255031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.131:46053 (local address 127.25.211.130:59399)
0806 02:57:03.255722 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0806 02:57:03.255739 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:57:03.255878 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:57:03.256304 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:57:03.256308 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:57:03.256341 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:57:03.256748 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:03.256758 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:03.258076 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:03.258079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:03.260347 (+ 2268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:03.260364 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:03.261611 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:03.261618 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:03.261796 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:03.261824 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 02:57:03.261901 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 02:57:03.262028 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"mutex_wait_us":1,"thread_start_us":350,"threads_started":1}
I20260806 02:57:03.262573 20289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:03.255770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:34237 (local address 127.25.211.130:55763)
0806 02:57:03.256448 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0806 02:57:03.256465 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:57:03.256591 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:57:03.256896 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:57:03.256899 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:57:03.256919 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:57:03.257183 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:03.257190 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:03.258436 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:03.258441 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:03.260472 (+ 2031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:03.260483 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:03.261695 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:03.261700 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:03.261844 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:03.261869 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 02:57:03.261962 (+ 93us) client_negotiation.cc:240] Negotiation successful
0806 02:57:03.262094 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"mutex_wait_us":25,"thread_start_us":359,"threads_started":1}
I20260806 02:57:03.263556 19803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21d22db7c2047049fa1dbd0688ffa6c" candidate_uuid: "48bd2ee03a684fbfa40afa51a907faa7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55a15f23e9ce445089da79455f1ff981" is_pre_election: true
I20260806 02:57:03.263793 20179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21d22db7c2047049fa1dbd0688ffa6c" candidate_uuid: "48bd2ee03a684fbfa40afa51a907faa7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad9b499e0c844724bc038bdc0cc37f74" is_pre_election: true
I20260806 02:57:03.264156 19803 raft_consensus.cc:2454] T a21d22db7c2047049fa1dbd0688ffa6c P 55a15f23e9ce445089da79455f1ff981 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48bd2ee03a684fbfa40afa51a907faa7 in term 0.
I20260806 02:57:03.264288 20179 raft_consensus.cc:2454] T a21d22db7c2047049fa1dbd0688ffa6c P ad9b499e0c844724bc038bdc0cc37f74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48bd2ee03a684fbfa40afa51a907faa7 in term 0.
I20260806 02:57:03.264918 19895 leader_election.cc:302] T a21d22db7c2047049fa1dbd0688ffa6c P 48bd2ee03a684fbfa40afa51a907faa7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48bd2ee03a684fbfa40afa51a907faa7, 55a15f23e9ce445089da79455f1ff981; no voters:
I20260806 02:57:03.265336 20282 raft_consensus.cc:2781] T a21d22db7c2047049fa1dbd0688ffa6c P 48bd2ee03a684fbfa40afa51a907faa7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:57:03.265496 20282 raft_consensus.cc:482] T a21d22db7c2047049fa1dbd0688ffa6c P 48bd2ee03a684fbfa40afa51a907faa7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:57:03.265638 20282 raft_consensus.cc:3037] T a21d22db7c2047049fa1dbd0688ffa6c P 48bd2ee03a684fbfa40afa51a907faa7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:57:03.267992 20282 raft_consensus.cc:504] T a21d22db7c2047049fa1dbd0688ffa6c P 48bd2ee03a684fbfa40afa51a907faa7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48bd2ee03a684fbfa40afa51a907faa7" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 35823 } } peers { permanent_uuid: "55a15f23e9ce445089da79455f1ff981" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 34237 } } peers { permanent_uuid: "ad9b499e0c844724bc038bdc0cc37f74" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 46053 } }
I20260806 02:57:03.268647 20282 leader_election.cc:288] T a21d22db7c2047049fa1dbd0688ffa6c P 48bd2ee03a684fbfa40afa51a907faa7 [CANDIDATE]: Term 1 election: Requested vote from peers 55a15f23e9ce445089da79455f1ff981 (127.25.211.129:34237), ad9b499e0c844724bc038bdc0cc37f74 (127.25.211.131:46053)
I20260806 02:57:03.269095 19803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21d22db7c2047049fa1dbd0688ffa6c" candidate_uuid: "48bd2ee03a684fbfa40afa51a907faa7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55a15f23e9ce445089da79455f1ff981"
I20260806 02:57:03.269173 20179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a21d22db7c2047049fa1dbd0688ffa6c" candidate_uuid: "48bd2ee03a684fbfa40afa51a907faa7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad9b499e0c844724bc038bdc0cc37f74"
I20260806 02:57:03.269284 19803 raft_consensus.cc:3037] T a21d22db7c2047049fa1dbd0688ffa6c P 55a15f23e9ce445089da79455f1ff981 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:57:03.269393 20179 raft_consensus.cc:3037] T a21d22db7c2047049fa1dbd0688ffa6c P ad9b499e0c844724bc038bdc0cc37f74 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:57:03.271677 19803 raft_consensus.cc:2454] T a21d22db7c2047049fa1dbd0688ffa6c P 55a15f23e9ce445089da79455f1ff981 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48bd2ee03a684fbfa40afa51a907faa7 in term 1.
I20260806 02:57:03.271694 20179 raft_consensus.cc:2454] T a21d22db7c2047049fa1dbd0688ffa6c P ad9b499e0c844724bc038bdc0cc37f74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48bd2ee03a684fbfa40afa51a907faa7 in term 1.
I20260806 02:57:03.272104 19895 leader_election.cc:302] T a21d22db7c2047049fa1dbd0688ffa6c P 48bd2ee03a684fbfa40afa51a907faa7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48bd2ee03a684fbfa40afa51a907faa7, 55a15f23e9ce445089da79455f1ff981; no voters:
I20260806 02:57:03.272418 20282 raft_consensus.cc:2781] T a21d22db7c2047049fa1dbd0688ffa6c P 48bd2ee03a684fbfa40afa51a907faa7 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:57:03.272873 20282 raft_consensus.cc:686] T a21d22db7c2047049fa1dbd0688ffa6c P 48bd2ee03a684fbfa40afa51a907faa7 [term 1 LEADER]: Becoming Leader. State: Replica: 48bd2ee03a684fbfa40afa51a907faa7, State: Running, Role: LEADER
I20260806 02:57:03.273430 20282 consensus_queue.cc:237] T a21d22db7c2047049fa1dbd0688ffa6c P 48bd2ee03a684fbfa40afa51a907faa7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48bd2ee03a684fbfa40afa51a907faa7" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 35823 } } peers { permanent_uuid: "55a15f23e9ce445089da79455f1ff981" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 34237 } } peers { permanent_uuid: "ad9b499e0c844724bc038bdc0cc37f74" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 46053 } }
W20260806 02:57:03.274303 20252 tablet.cc:2369] T a21d22db7c2047049fa1dbd0688ffa6c P ad9b499e0c844724bc038bdc0cc37f74: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:57:03.278306 19599 catalog_manager.cc:5495] T a21d22db7c2047049fa1dbd0688ffa6c P 48bd2ee03a684fbfa40afa51a907faa7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 48bd2ee03a684fbfa40afa51a907faa7 (127.25.211.130). New cstate: current_term: 1 leader_uuid: "48bd2ee03a684fbfa40afa51a907faa7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48bd2ee03a684fbfa40afa51a907faa7" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 35823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55a15f23e9ce445089da79455f1ff981" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 34237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad9b499e0c844724bc038bdc0cc37f74" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 46053 } health_report { overall_health: UNKNOWN } } }
I20260806 02:57:03.299414 20272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:03.293899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38252 (local address 127.25.211.130:35823)
0806 02:57:03.294121 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:03.294126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:03.294145 (+ 19us) server_negotiation.cc:407] Connection header received
0806 02:57:03.294394 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:03.294398 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:03.294477 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:03.294614 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:57:03.294942 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:03.295716 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:03.297244 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:03.297686 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:03.297809 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:03.298299 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:57:03.298818 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:57:03.298927 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:03.299063 (+ 136us) server_negotiation.cc:299] Negotiation successful
0806 02:57:03.299136 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260806 02:57:03.307742 20179 raft_consensus.cc:1261] T a21d22db7c2047049fa1dbd0688ffa6c P ad9b499e0c844724bc038bdc0cc37f74 [term 1 FOLLOWER]: Refusing update from remote peer 48bd2ee03a684fbfa40afa51a907faa7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:57:03.307742 19803 raft_consensus.cc:1261] T a21d22db7c2047049fa1dbd0688ffa6c P 55a15f23e9ce445089da79455f1ff981 [term 1 FOLLOWER]: Refusing update from remote peer 48bd2ee03a684fbfa40afa51a907faa7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:57:03.308697 20291 consensus_queue.cc:1035] T a21d22db7c2047049fa1dbd0688ffa6c P 48bd2ee03a684fbfa40afa51a907faa7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55a15f23e9ce445089da79455f1ff981" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 34237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:57:03.309023 20282 consensus_queue.cc:1035] T a21d22db7c2047049fa1dbd0688ffa6c P 48bd2ee03a684fbfa40afa51a907faa7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad9b499e0c844724bc038bdc0cc37f74" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 46053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:57:03.319624 20294 mvcc.cc:205] Tried to move back new op lower bound from 7315394655458611200 to 7315394655332196352. Current Snapshot: MvccSnapshot[applied={T|T < 7315394655458611200}]
I20260806 02:57:03.341925 19599 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:57:03.342236 19599 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:57:03.346128 19599 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b8b5a4bebd144b038472d34987e3088c: Sending DeleteTablet for 3 replicas of tablet a21d22db7c2047049fa1dbd0688ffa6c
I20260806 02:57:03.347482 19775 tablet_service.cc:1514] Processing DeleteTablet for tablet a21d22db7c2047049fa1dbd0688ffa6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:57:03 UTC) from {username='kudu', principal='kudu/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:34244
I20260806 02:57:03.347501 19964 tablet_service.cc:1514] Processing DeleteTablet for tablet a21d22db7c2047049fa1dbd0688ffa6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:57:03 UTC) from {username='kudu', principal='kudu/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:38238
I20260806 02:57:03.347568 20153 tablet_service.cc:1514] Processing DeleteTablet for tablet a21d22db7c2047049fa1dbd0688ffa6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:57:03 UTC) from {username='kudu', principal='kudu/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:33516
I20260806 02:57:03.349076 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 19686
I20260806 02:57:03.360704 20314 tablet_replica.cc:330] T a21d22db7c2047049fa1dbd0688ffa6c P 55a15f23e9ce445089da79455f1ff981: stopping tablet replica
I20260806 02:57:03.361434 20314 raft_consensus.cc:2229] T a21d22db7c2047049fa1dbd0688ffa6c P 55a15f23e9ce445089da79455f1ff981 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:57:03.362090 20314 raft_consensus.cc:2258] T a21d22db7c2047049fa1dbd0688ffa6c P 55a15f23e9ce445089da79455f1ff981 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:57:03.365373 20312 tablet_replica.cc:330] T a21d22db7c2047049fa1dbd0688ffa6c P ad9b499e0c844724bc038bdc0cc37f74: stopping tablet replica
I20260806 02:57:03.365893 20312 raft_consensus.cc:2229] T a21d22db7c2047049fa1dbd0688ffa6c P ad9b499e0c844724bc038bdc0cc37f74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:57:03.366358 20314 ts_tablet_manager.cc:1905] T a21d22db7c2047049fa1dbd0688ffa6c P 55a15f23e9ce445089da79455f1ff981: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:57:03.366465 20312 raft_consensus.cc:2258] T a21d22db7c2047049fa1dbd0688ffa6c P ad9b499e0c844724bc038bdc0cc37f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:57:03.367766 20313 tablet_replica.cc:330] T a21d22db7c2047049fa1dbd0688ffa6c P 48bd2ee03a684fbfa40afa51a907faa7: stopping tablet replica
I20260806 02:57:03.368274 20313 raft_consensus.cc:2229] T a21d22db7c2047049fa1dbd0688ffa6c P 48bd2ee03a684fbfa40afa51a907faa7 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:57:03.368913 20312 ts_tablet_manager.cc:1905] T a21d22db7c2047049fa1dbd0688ffa6c P ad9b499e0c844724bc038bdc0cc37f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:57:03.369019 20313 raft_consensus.cc:2258] T a21d22db7c2047049fa1dbd0688ffa6c P 48bd2ee03a684fbfa40afa51a907faa7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:57:03.371243 20313 ts_tablet_manager.cc:1905] T a21d22db7c2047049fa1dbd0688ffa6c P 48bd2ee03a684fbfa40afa51a907faa7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:57:03.374193 20314 ts_tablet_manager.cc:1918] T a21d22db7c2047049fa1dbd0688ffa6c P 55a15f23e9ce445089da79455f1ff981: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:57:03.374420 20314 log.cc:1196] T a21d22db7c2047049fa1dbd0688ffa6c P 55a15f23e9ce445089da79455f1ff981: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/ts-0/wal/wals/a21d22db7c2047049fa1dbd0688ffa6c
I20260806 02:57:03.375026 20314 ts_tablet_manager.cc:1939] T a21d22db7c2047049fa1dbd0688ffa6c P 55a15f23e9ce445089da79455f1ff981: Deleting consensus metadata
I20260806 02:57:03.376013 20312 ts_tablet_manager.cc:1918] T a21d22db7c2047049fa1dbd0688ffa6c P ad9b499e0c844724bc038bdc0cc37f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:57:03.376240 20312 log.cc:1196] T a21d22db7c2047049fa1dbd0688ffa6c P ad9b499e0c844724bc038bdc0cc37f74: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/ts-2/wal/wals/a21d22db7c2047049fa1dbd0688ffa6c
I20260806 02:57:03.376828 20312 ts_tablet_manager.cc:1939] T a21d22db7c2047049fa1dbd0688ffa6c P ad9b499e0c844724bc038bdc0cc37f74: Deleting consensus metadata
I20260806 02:57:03.377261 19579 catalog_manager.cc:4841] TS 55a15f23e9ce445089da79455f1ff981 (127.25.211.129:34237): tablet a21d22db7c2047049fa1dbd0688ffa6c (table test-table [id=723f1d77b106444c9c6a03568eb16314]) successfully deleted
I20260806 02:57:03.378791 19577 catalog_manager.cc:4841] TS ad9b499e0c844724bc038bdc0cc37f74 (127.25.211.131:46053): tablet a21d22db7c2047049fa1dbd0688ffa6c (table test-table [id=723f1d77b106444c9c6a03568eb16314]) successfully deleted
I20260806 02:57:03.380066 20313 ts_tablet_manager.cc:1918] T a21d22db7c2047049fa1dbd0688ffa6c P 48bd2ee03a684fbfa40afa51a907faa7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:57:03.380250 20313 log.cc:1196] T a21d22db7c2047049fa1dbd0688ffa6c P 48bd2ee03a684fbfa40afa51a907faa7: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1785984707115702-26446-0/minicluster-data/ts-1/wal/wals/a21d22db7c2047049fa1dbd0688ffa6c
Aug 06 02:57:03 dist-test-slave-zj8q krb5kdc[19536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.129@KRBTEST.COM
I20260806 02:57:03.380856 20313 ts_tablet_manager.cc:1939] T a21d22db7c2047049fa1dbd0688ffa6c P 48bd2ee03a684fbfa40afa51a907faa7: Deleting consensus metadata
I20260806 02:57:03.382820 19577 catalog_manager.cc:4841] TS 48bd2ee03a684fbfa40afa51a907faa7 (127.25.211.130:35823): tablet a21d22db7c2047049fa1dbd0688ffa6c (table test-table [id=723f1d77b106444c9c6a03568eb16314]) successfully deleted
I20260806 02:57:03.388053 20273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:03.350640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34252 (local address 127.25.211.129:34237)
0806 02:57:03.350834 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:03.350840 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:03.361391 (+ 10551us) server_negotiation.cc:407] Connection header received
0806 02:57:03.361803 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:03.361809 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:03.361912 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:03.362005 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:03.364421 (+ 2416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:03.365563 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:03.367710 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:03.368409 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:03.368525 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:03.381750 (+ 13225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:03.381766 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:03.381782 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:57:03.381822 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:03.384900 (+ 3078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:03.385851 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:03.385858 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:03.385860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:03.386000 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:03.386462 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:03.386465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:03.386467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:03.386873 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:03.387097 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:03.387543 (+ 446us) server_negotiation.cc:299] Negotiation successful
0806 02:57:03.387736 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 02:57:03.388741 19861 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:57:03.498745 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 19686
I20260806 02:57:03.523089 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 19874
Aug 06 02:57:03 dist-test-slave-zj8q krb5kdc[19536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.130@KRBTEST.COM
I20260806 02:57:03.539650 20272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:03.523870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38268 (local address 127.25.211.130:35823)
0806 02:57:03.524060 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:03.524069 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:03.524134 (+ 65us) server_negotiation.cc:407] Connection header received
0806 02:57:03.524328 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:03.524332 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:03.524412 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:03.524504 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:03.525938 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:03.526863 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:03.528192 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:03.528630 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:03.528751 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:03.533398 (+ 4647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:03.533416 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:03.533435 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:57:03.533480 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:03.536597 (+ 3117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:03.537337 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:03.537346 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:03.537348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:03.537497 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:03.537880 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:03.537883 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:03.537886 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:03.538441 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:03.538803 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:03.539057 (+ 254us) server_negotiation.cc:299] Negotiation successful
0806 02:57:03.539305 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260806 02:57:03.540416 20052 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:57:03.654089 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 19874
I20260806 02:57:03.677501 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 20070
Aug 06 02:57:03 dist-test-slave-zj8q krb5kdc[19536](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.131@KRBTEST.COM
I20260806 02:57:03.693609 20275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:03.678329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33532 (local address 127.25.211.131:46053)
0806 02:57:03.678515 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:03.678520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:03.678540 (+ 20us) server_negotiation.cc:407] Connection header received
0806 02:57:03.678700 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:03.678704 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:03.678780 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:03.678861 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:03.680211 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:03.680955 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:03.682209 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:03.682682 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:03.682778 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:03.687422 (+ 4644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:03.687439 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:03.687459 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:57:03.687513 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:03.691165 (+ 3652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:03.691829 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:03.691836 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:03.691838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:03.691983 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:03.692278 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:03.692281 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:03.692283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:03.692634 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:03.692859 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:03.693115 (+ 256us) server_negotiation.cc:299] Negotiation successful
0806 02:57:03.693282 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 02:57:03.694366 20240 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:57:03.782686 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 20070
I20260806 02:57:03.805783 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 19555
I20260806 02:57:03.807057 19643 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:57:03.917317 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 19555
2026-08-06T02:57:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4728 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-06T02:57:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:57:03Z Disabled control of system clock
I20260806 02:57:04.019922 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:44373
--webserver_interface=127.25.211.190
--webserver_port=0
--builtin_ntp_servers=127.25.211.148:45075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.211.190:44373
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 02:57:04.369871 20327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:57:04.370245 20327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:57:04.370386 20327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:57:04.377739 20327 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:57:04.377866 20327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:57:04.377910 20327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:57:04.378002 20327 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:57:04.378059 20327 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:57:04.388475 20327 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:45075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.211.190:44373
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:44373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.211.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:57:04.389910 20327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:57:04.391865 20327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:57:04.403743 20335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:04.403769 20340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:04.403978 20336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:57:04.404898 20327 server_base.cc:1034] running on GCE node
I20260806 02:57:04.405774 20327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:57:04.407289 20327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:57:04.408499 20327 hybrid_clock.cc:648] HybridClock initialized: now 1785985024408448 us; error 46 us; skew 500 ppm
I20260806 02:57:04.408846 20327 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:57:04.411247 20327 webserver.cc:460] Webserver started at http://127.25.211.190:39625/ using document root <none> and password file <none>
I20260806 02:57:04.411957 20327 fs_manager.cc:359] Metadata directory not provided
I20260806 02:57:04.412074 20327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:57:04.412400 20327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:57:04.415634 20327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/master-0/data/instance:
uuid: "5d42ef1ebb93421fad1f9fb37499410f"
format_stamp: "Formatted at 2026-08-06 02:57:04 on dist-test-slave-zj8q"
server_key: "6cc7d88e89f957fa61457e45b455f3b8"
server_key_iv: "0d81b3823c7a1f63b11d89f85170d73b"
server_key_version: "encryptionkey@0"
I20260806 02:57:04.416522 20327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/master-0/wal/instance:
uuid: "5d42ef1ebb93421fad1f9fb37499410f"
format_stamp: "Formatted at 2026-08-06 02:57:04 on dist-test-slave-zj8q"
server_key: "6cc7d88e89f957fa61457e45b455f3b8"
server_key_iv: "0d81b3823c7a1f63b11d89f85170d73b"
server_key_version: "encryptionkey@0"
I20260806 02:57:04.422278 20327 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260806 02:57:04.425864 20348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:04.427345 20327 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:57:04.427531 20327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/master-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/master-0/wal
uuid: "5d42ef1ebb93421fad1f9fb37499410f"
format_stamp: "Formatted at 2026-08-06 02:57:04 on dist-test-slave-zj8q"
server_key: "6cc7d88e89f957fa61457e45b455f3b8"
server_key_iv: "0d81b3823c7a1f63b11d89f85170d73b"
server_key_version: "encryptionkey@0"
I20260806 02:57:04.427765 20327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:57:04.441390 20327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:57:04.442301 20327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:57:04.442579 20327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:57:04.459798 20327 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.190:44373
I20260806 02:57:04.459836 20425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.190:44373 every 8 connection(s)
I20260806 02:57:04.461115 20327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
I20260806 02:57:04.462141 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 20327
I20260806 02:57:04.462358 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/master-0/wal/instance
I20260806 02:57:04.462857 26446 external_mini_cluster.cc:1428] Setting key 46edf2a4a3d37dd04b6f546f9e7fd992
I20260806 02:57:04.465588 20426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:57:04.476209 20432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:04.464836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36758 (local address 127.25.211.190:44373)
0806 02:57:04.465900 (+ 1064us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:04.465911 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:04.465955 (+ 44us) server_negotiation.cc:407] Connection header received
0806 02:57:04.467598 (+ 1643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:04.467620 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:04.468111 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:04.468504 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:04.469060 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:04.470478 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:04.471865 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:04.472539 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:04.472714 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:04.473863 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:04.473874 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:04.473888 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:57:04.474505 (+ 617us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:04.474599 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:57:04.474622 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:04.474734 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:04.474962 (+ 228us) server_negotiation.cc:299] Negotiation successful
0806 02:57:04.475267 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":354,"threads_started":1}
I20260806 02:57:04.476643 20426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d42ef1ebb93421fad1f9fb37499410f: Bootstrap starting.
I20260806 02:57:04.480245 20426 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d42ef1ebb93421fad1f9fb37499410f: Neither blocks nor log segments found. Creating new log.
I20260806 02:57:04.481426 20426 log.cc:824] T 00000000000000000000000000000000 P 5d42ef1ebb93421fad1f9fb37499410f: Log is configured to *not* fsync() on all Append() calls
I20260806 02:57:04.484076 20426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d42ef1ebb93421fad1f9fb37499410f: No bootstrap required, opened a new log
I20260806 02:57:04.491560 20426 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d42ef1ebb93421fad1f9fb37499410f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d42ef1ebb93421fad1f9fb37499410f" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 44373 } }
I20260806 02:57:04.491914 20426 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d42ef1ebb93421fad1f9fb37499410f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:57:04.491988 20426 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d42ef1ebb93421fad1f9fb37499410f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d42ef1ebb93421fad1f9fb37499410f, State: Initialized, Role: FOLLOWER
I20260806 02:57:04.492478 20426 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d42ef1ebb93421fad1f9fb37499410f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d42ef1ebb93421fad1f9fb37499410f" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 44373 } }
I20260806 02:57:04.492676 20426 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d42ef1ebb93421fad1f9fb37499410f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:57:04.492770 20426 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d42ef1ebb93421fad1f9fb37499410f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:57:04.492898 20426 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d42ef1ebb93421fad1f9fb37499410f [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:57:04.495420 20426 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d42ef1ebb93421fad1f9fb37499410f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d42ef1ebb93421fad1f9fb37499410f" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 44373 } }
I20260806 02:57:04.495872 20426 leader_election.cc:302] T 00000000000000000000000000000000 P 5d42ef1ebb93421fad1f9fb37499410f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d42ef1ebb93421fad1f9fb37499410f; no voters:
I20260806 02:57:04.496471 20426 leader_election.cc:288] T 00000000000000000000000000000000 P 5d42ef1ebb93421fad1f9fb37499410f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:57:04.496627 20435 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d42ef1ebb93421fad1f9fb37499410f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:57:04.497294 20435 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d42ef1ebb93421fad1f9fb37499410f [term 1 LEADER]: Becoming Leader. State: Replica: 5d42ef1ebb93421fad1f9fb37499410f, State: Running, Role: LEADER
I20260806 02:57:04.497782 20435 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d42ef1ebb93421fad1f9fb37499410f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d42ef1ebb93421fad1f9fb37499410f" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 44373 } }
I20260806 02:57:04.498471 20426 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d42ef1ebb93421fad1f9fb37499410f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:57:04.500492 20438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d42ef1ebb93421fad1f9fb37499410f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d42ef1ebb93421fad1f9fb37499410f. Latest consensus state: current_term: 1 leader_uuid: "5d42ef1ebb93421fad1f9fb37499410f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d42ef1ebb93421fad1f9fb37499410f" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 44373 } } }
I20260806 02:57:04.500696 20436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d42ef1ebb93421fad1f9fb37499410f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d42ef1ebb93421fad1f9fb37499410f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d42ef1ebb93421fad1f9fb37499410f" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 44373 } } }
I20260806 02:57:04.500949 20438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d42ef1ebb93421fad1f9fb37499410f [sys.catalog]: This master's current role is: LEADER
I20260806 02:57:04.500952 20436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d42ef1ebb93421fad1f9fb37499410f [sys.catalog]: This master's current role is: LEADER
I20260806 02:57:04.502182 20445 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:57:04.509433 20445 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:57:04.519287 20445 catalog_manager.cc:1347] Generated new cluster ID: 9b13335ea30d43b49d6225347ab6a290
I20260806 02:57:04.519419 20445 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:57:04.538311 20445 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:57:04.539570 20445 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:57:04.547047 20445 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d42ef1ebb93421fad1f9fb37499410f: Generated new TSK 0
I20260806 02:57:04.547894 20445 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:57:04.567651 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.129:0
--local_ip_for_outbound_sockets=127.25.211.129
--webserver_interface=127.25.211.129
--webserver_port=0
--tserver_master_addrs=127.25.211.190:44373
--builtin_ntp_servers=127.25.211.148:45075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 02:57:04.912002 20459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:57:04.912281 20459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:57:04.912366 20459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:57:04.919495 20459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:57:04.919601 20459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:57:04.919706 20459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.129
I20260806 02:57:04.928694 20459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:45075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.211.129
--webserver_port=0
--tserver_master_addrs=127.25.211.190:44373
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.129
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:57:04.929912 20459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:57:04.931896 20459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:57:04.944679 20467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:04.944703 20468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:04.944888 20470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:57:04.945655 20459 server_base.cc:1034] running on GCE node
I20260806 02:57:04.946445 20459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:57:04.947405 20459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:57:04.948608 20459 hybrid_clock.cc:648] HybridClock initialized: now 1785985024948559 us; error 44 us; skew 500 ppm
I20260806 02:57:04.948920 20459 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:57:04.951239 20459 webserver.cc:460] Webserver started at http://127.25.211.129:46013/ using document root <none> and password file <none>
I20260806 02:57:04.951929 20459 fs_manager.cc:359] Metadata directory not provided
I20260806 02:57:04.952023 20459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:57:04.952358 20459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:57:04.955328 20459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/ts-0/data/instance:
uuid: "0548a692f63442538f70b686f25beba6"
format_stamp: "Formatted at 2026-08-06 02:57:04 on dist-test-slave-zj8q"
server_key: "68b68fe1f3e296d9774421095705470d"
server_key_iv: "71c4f5157bb79f8727599315b83d825f"
server_key_version: "encryptionkey@0"
I20260806 02:57:04.956118 20459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/ts-0/wal/instance:
uuid: "0548a692f63442538f70b686f25beba6"
format_stamp: "Formatted at 2026-08-06 02:57:04 on dist-test-slave-zj8q"
server_key: "68b68fe1f3e296d9774421095705470d"
server_key_iv: "71c4f5157bb79f8727599315b83d825f"
server_key_version: "encryptionkey@0"
I20260806 02:57:04.961663 20459 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260806 02:57:04.965399 20478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:04.966890 20459 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:57:04.967053 20459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/ts-0/wal
uuid: "0548a692f63442538f70b686f25beba6"
format_stamp: "Formatted at 2026-08-06 02:57:04 on dist-test-slave-zj8q"
server_key: "68b68fe1f3e296d9774421095705470d"
server_key_iv: "71c4f5157bb79f8727599315b83d825f"
server_key_version: "encryptionkey@0"
I20260806 02:57:04.967276 20459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:57:04.981382 20459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:57:04.982281 20459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:57:04.982556 20459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:57:04.983614 20459 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:57:04.985080 20459 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:57:04.985167 20459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:04.985250 20459 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:57:04.985303 20459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:05.017390 20459 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.129:38215
I20260806 02:57:05.017437 20645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.129:38215 every 8 connection(s)
I20260806 02:57:05.018783 20459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
I20260806 02:57:05.023916 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 20459
I20260806 02:57:05.024142 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/ts-0/wal/instance
I20260806 02:57:05.024717 26446 external_mini_cluster.cc:1428] Setting key 429ca5cbd9c8bcf35d6e0b237d2f6d27
I20260806 02:57:05.028160 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.130:0
--local_ip_for_outbound_sockets=127.25.211.130
--webserver_interface=127.25.211.130
--webserver_port=0
--tserver_master_addrs=127.25.211.190:44373
--builtin_ntp_servers=127.25.211.148:45075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 02:57:05.033613 20650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:05.021401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:34603 (local address 127.25.211.190:44373)
0806 02:57:05.021840 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:05.021844 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:05.023000 (+ 1156us) server_negotiation.cc:407] Connection header received
0806 02:57:05.025465 (+ 2465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:05.025472 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:05.025580 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:05.025700 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:05.027253 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:05.028285 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:05.030655 (+ 2370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:05.031068 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:05.031178 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:05.032845 (+ 1667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:05.032851 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:05.032854 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:57:05.032875 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:05.032922 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:57:05.032935 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:05.033004 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:05.033248 (+ 244us) server_negotiation.cc:299] Negotiation successful
0806 02:57:05.033340 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":233,"threads_started":1}
I20260806 02:57:05.035180 20651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:05.021867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:44373 (local address 127.25.211.129:34603)
0806 02:57:05.022830 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0806 02:57:05.022875 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 02:57:05.025108 (+ 2233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:57:05.026005 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:57:05.026017 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:57:05.026201 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:57:05.027050 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:05.027092 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:05.028484 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:05.028489 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:05.030472 (+ 1983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:05.030488 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:05.031310 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:05.031317 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:05.031619 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:05.032432 (+ 813us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:57:05.032435 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:57:05.032492 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:57:05.032612 (+ 120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:57:05.033024 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:57:05.033035 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:57:05.033044 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 02:57:05.033224 (+ 180us) client_negotiation.cc:240] Negotiation successful
0806 02:57:05.033665 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":329,"threads_started":1}
I20260806 02:57:05.036959 20647 heartbeater.cc:347] Connected to a master server at 127.25.211.190:44373
I20260806 02:57:05.037292 20647 heartbeater.cc:464] Registering TS with master...
I20260806 02:57:05.038089 20647 heartbeater.cc:511] Master 127.25.211.190:44373 requested a full tablet report, sending...
I20260806 02:57:05.040278 20376 ts_manager.cc:194] Registered new tserver with Master: 0548a692f63442538f70b686f25beba6 (127.25.211.129:38215)
I20260806 02:57:05.041813 20376 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.211.129:34603
W20260806 02:57:05.367630 20652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:57:05.367966 20652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:57:05.368116 20652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:57:05.375519 20652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:57:05.375661 20652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:57:05.375800 20652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.130
I20260806 02:57:05.385170 20652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:45075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.211.130
--webserver_port=0
--tserver_master_addrs=127.25.211.190:44373
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.130
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:57:05.386603 20652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:57:05.388381 20652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:57:05.401603 20661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:05.401680 20664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:05.401620 20669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:57:05.402848 20652 server_base.cc:1034] running on GCE node
I20260806 02:57:05.403496 20652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:57:05.404454 20652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:57:05.405661 20652 hybrid_clock.cc:648] HybridClock initialized: now 1785985025405607 us; error 43 us; skew 500 ppm
I20260806 02:57:05.405980 20652 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:57:05.408401 20652 webserver.cc:460] Webserver started at http://127.25.211.130:42333/ using document root <none> and password file <none>
I20260806 02:57:05.409111 20652 fs_manager.cc:359] Metadata directory not provided
I20260806 02:57:05.409206 20652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:57:05.409518 20652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:57:05.412532 20652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/ts-1/data/instance:
uuid: "333b1585ca0a47f2b268452c6396af54"
format_stamp: "Formatted at 2026-08-06 02:57:05 on dist-test-slave-zj8q"
server_key: "07ff691ae8475d2d65001f6481a25b20"
server_key_iv: "64bcfec8565b16030dbe686f74b4fe71"
server_key_version: "encryptionkey@0"
I20260806 02:57:05.413333 20652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/ts-1/wal/instance:
uuid: "333b1585ca0a47f2b268452c6396af54"
format_stamp: "Formatted at 2026-08-06 02:57:05 on dist-test-slave-zj8q"
server_key: "07ff691ae8475d2d65001f6481a25b20"
server_key_iv: "64bcfec8565b16030dbe686f74b4fe71"
server_key_version: "encryptionkey@0"
I20260806 02:57:05.418622 20652 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260806 02:57:05.422049 20680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:05.423467 20652 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:57:05.423640 20652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/ts-1/wal
uuid: "333b1585ca0a47f2b268452c6396af54"
format_stamp: "Formatted at 2026-08-06 02:57:05 on dist-test-slave-zj8q"
server_key: "07ff691ae8475d2d65001f6481a25b20"
server_key_iv: "64bcfec8565b16030dbe686f74b4fe71"
server_key_version: "encryptionkey@0"
I20260806 02:57:05.423902 20652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:57:05.449349 20652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:57:05.450305 20652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:57:05.450621 20652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:57:05.451735 20652 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:57:05.453269 20652 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:57:05.453367 20652 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:05.453481 20652 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:57:05.453557 20652 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:05.489271 20652 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.130:38591
I20260806 02:57:05.489349 20852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.130:38591 every 8 connection(s)
I20260806 02:57:05.490603 20652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
I20260806 02:57:05.492523 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 20652
I20260806 02:57:05.492709 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/ts-1/wal/instance
I20260806 02:57:05.493178 26446 external_mini_cluster.cc:1428] Setting key 2dd54330c26d77074f2a354eab88710a
I20260806 02:57:05.496090 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.131:0
--local_ip_for_outbound_sockets=127.25.211.131
--webserver_interface=127.25.211.131
--webserver_port=0
--tserver_master_addrs=127.25.211.190:44373
--builtin_ntp_servers=127.25.211.148:45075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 02:57:05.504047 20650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:05.493328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:39543 (local address 127.25.211.190:44373)
0806 02:57:05.493540 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:05.493546 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:05.495180 (+ 1634us) server_negotiation.cc:407] Connection header received
0806 02:57:05.497050 (+ 1870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:05.497057 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:05.497142 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:05.497246 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:05.498345 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:05.499391 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:05.500990 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:05.501509 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:05.501635 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:05.503093 (+ 1458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:05.503099 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:05.503103 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:57:05.503157 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:05.503196 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:57:05.503212 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:05.503297 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:05.503568 (+ 271us) server_negotiation.cc:299] Negotiation successful
0806 02:57:05.503680 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 02:57:05.505206 20856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:05.493907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:44373 (local address 127.25.211.130:39543)
0806 02:57:05.495016 (+ 1109us) negotiation.cc:107] Waiting for socket to connect
0806 02:57:05.495063 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 02:57:05.496765 (+ 1702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:57:05.497444 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:57:05.497452 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:57:05.497548 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:57:05.498118 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:05.498136 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:05.499540 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:05.499544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:05.500837 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:05.500851 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:05.501782 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:05.501788 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:05.502011 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:05.502751 (+ 740us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:57:05.502754 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:57:05.502799 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:57:05.502894 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:57:05.503297 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:57:05.503307 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:57:05.503316 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 02:57:05.503540 (+ 224us) client_negotiation.cc:240] Negotiation successful
0806 02:57:05.503939 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":340,"threads_started":1}
I20260806 02:57:05.506958 20854 heartbeater.cc:347] Connected to a master server at 127.25.211.190:44373
I20260806 02:57:05.507282 20854 heartbeater.cc:464] Registering TS with master...
I20260806 02:57:05.508087 20854 heartbeater.cc:511] Master 127.25.211.190:44373 requested a full tablet report, sending...
I20260806 02:57:05.509817 20376 ts_manager.cc:194] Registered new tserver with Master: 333b1585ca0a47f2b268452c6396af54 (127.25.211.130:38591)
I20260806 02:57:05.510928 20376 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.211.130:39543
W20260806 02:57:05.857160 20857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:57:05.857466 20857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:57:05.857583 20857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:57:05.865497 20857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:57:05.865655 20857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:57:05.865816 20857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.131
I20260806 02:57:05.875473 20857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:45075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.211.131
--webserver_port=0
--tserver_master_addrs=127.25.211.190:44373
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.131
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:57:05.876813 20857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:57:05.878726 20857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:57:05.892550 20872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:05.892459 20871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:05.892495 20874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:57:05.895737 20857 server_base.cc:1034] running on GCE node
I20260806 02:57:05.896446 20857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:57:05.897518 20857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:57:05.898759 20857 hybrid_clock.cc:648] HybridClock initialized: now 1785985025898692 us; error 58 us; skew 500 ppm
I20260806 02:57:05.899134 20857 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:57:05.901506 20857 webserver.cc:460] Webserver started at http://127.25.211.131:42043/ using document root <none> and password file <none>
I20260806 02:57:05.902308 20857 fs_manager.cc:359] Metadata directory not provided
I20260806 02:57:05.902424 20857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:57:05.902761 20857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:57:05.905820 20857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/ts-2/data/instance:
uuid: "5803b20899584cf48dc1679fcaff801d"
format_stamp: "Formatted at 2026-08-06 02:57:05 on dist-test-slave-zj8q"
server_key: "902872b86a9facf575fa0c48964742cc"
server_key_iv: "b3386106600d5fe09e3530d9a51840b6"
server_key_version: "encryptionkey@0"
I20260806 02:57:05.906664 20857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/ts-2/wal/instance:
uuid: "5803b20899584cf48dc1679fcaff801d"
format_stamp: "Formatted at 2026-08-06 02:57:05 on dist-test-slave-zj8q"
server_key: "902872b86a9facf575fa0c48964742cc"
server_key_iv: "b3386106600d5fe09e3530d9a51840b6"
server_key_version: "encryptionkey@0"
I20260806 02:57:05.911886 20857 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 02:57:05.915699 20884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:05.917217 20857 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 02:57:05.917375 20857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/ts-2/wal
uuid: "5803b20899584cf48dc1679fcaff801d"
format_stamp: "Formatted at 2026-08-06 02:57:05 on dist-test-slave-zj8q"
server_key: "902872b86a9facf575fa0c48964742cc"
server_key_iv: "b3386106600d5fe09e3530d9a51840b6"
server_key_version: "encryptionkey@0"
I20260806 02:57:05.917568 20857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:57:05.937551 20857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:57:05.938436 20857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:57:05.938719 20857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:57:05.939677 20857 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:57:05.941183 20857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:57:05.941277 20857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:05.941357 20857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:57:05.941409 20857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:05.971655 20857 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.131:37393
I20260806 02:57:05.971724 21050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.131:37393 every 8 connection(s)
I20260806 02:57:05.973026 20857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
I20260806 02:57:05.981535 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 20857
I20260806 02:57:05.981709 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/ts-2/wal/instance
I20260806 02:57:05.982190 26446 external_mini_cluster.cc:1428] Setting key ba02589240b586df5fd02662bc6d68e6
I20260806 02:57:05.985981 20650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:05.975663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:45825 (local address 127.25.211.190:44373)
0806 02:57:05.975839 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:05.975844 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:05.977288 (+ 1444us) server_negotiation.cc:407] Connection header received
0806 02:57:05.979153 (+ 1865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:05.979159 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:05.979254 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:05.979347 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:05.980481 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:05.981329 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:05.983306 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:05.983727 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:05.983842 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:05.985189 (+ 1347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:05.985194 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:05.985198 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:57:05.985215 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:05.985247 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:57:05.985258 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:05.985330 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:05.985558 (+ 228us) server_negotiation.cc:299] Negotiation successful
0806 02:57:05.985664 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 02:57:05.987236 21054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:05.976187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:44373 (local address 127.25.211.131:45825)
0806 02:57:05.977116 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0806 02:57:05.977161 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 02:57:05.978867 (+ 1706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:57:05.979598 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:57:05.979606 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:57:05.979698 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:57:05.980305 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:05.980322 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:05.981524 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:05.981529 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:05.983155 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:05.983170 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:05.983957 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:05.983965 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:05.984198 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:05.984857 (+ 659us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:57:05.984868 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:57:05.984914 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:57:05.985004 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:57:05.985347 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:57:05.985359 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:57:05.985366 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 02:57:05.985540 (+ 174us) client_negotiation.cc:240] Negotiation successful
0806 02:57:05.985985 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":275,"threads_started":1}
I20260806 02:57:05.988596 21051 heartbeater.cc:347] Connected to a master server at 127.25.211.190:44373
I20260806 02:57:05.988890 21051 heartbeater.cc:464] Registering TS with master...
I20260806 02:57:05.989647 21051 heartbeater.cc:511] Master 127.25.211.190:44373 requested a full tablet report, sending...
I20260806 02:57:05.991076 20376 ts_manager.cc:194] Registered new tserver with Master: 5803b20899584cf48dc1679fcaff801d (127.25.211.131:37393)
I20260806 02:57:05.991865 20376 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.211.131:45825
I20260806 02:57:05.997061 26446 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:57:06.007637 20650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:06.001399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36772 (local address 127.25.211.190:44373)
0806 02:57:06.001659 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:06.001666 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:06.002096 (+ 430us) server_negotiation.cc:407] Connection header received
0806 02:57:06.002254 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:06.002258 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:06.002339 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:06.002425 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:06.002920 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:06.003719 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:06.005201 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:06.005665 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:06.005766 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:06.006931 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:06.006938 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:06.006943 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:57:06.006967 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:06.007000 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:57:06.007015 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:06.007106 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:06.007247 (+ 141us) server_negotiation.cc:299] Negotiation successful
0806 02:57:06.007350 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 02:57:06.011570 20376 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36772:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:57:06.020865 20376 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:57:06.045411 20647 heartbeater.cc:503] Master 127.25.211.190:44373 was elected leader, sending a full tablet report...
I20260806 02:57:06.059580 21073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:06.033657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.130:38591 (local address 127.0.0.1:60868)
0806 02:57:06.046135 (+ 12478us) negotiation.cc:107] Waiting for socket to connect
0806 02:57:06.046224 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0806 02:57:06.046430 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:57:06.051093 (+ 4663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:57:06.051099 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:57:06.051132 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:57:06.051440 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:06.051450 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:06.053492 (+ 2042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:06.053501 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:06.056512 (+ 3011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:06.056536 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:06.058560 (+ 2024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:06.058568 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:06.058814 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:06.058870 (+ 56us) client_negotiation.cc:769] Sending connection context
0806 02:57:06.059015 (+ 145us) client_negotiation.cc:240] Negotiation successful
0806 02:57:06.059222 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12149,"thread_start_us":14296,"threads_started":1}
I20260806 02:57:06.059633 21072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:06.033678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60868 (local address 127.25.211.130:38591)
0806 02:57:06.050081 (+ 16403us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:06.050091 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:06.050123 (+ 32us) server_negotiation.cc:407] Connection header received
0806 02:57:06.050271 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:06.050280 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:06.050579 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:06.050798 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:57:06.051630 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:06.053295 (+ 1665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:06.056771 (+ 3476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:06.058341 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:06.058509 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:06.058762 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:06.059087 (+ 325us) server_negotiation.cc:299] Negotiation successful
0806 02:57:06.059280 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16216,"thread_start_us":17900,"threads_started":1}
I20260806 02:57:06.060735 21075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:06.033755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.131:37393 (local address 127.0.0.1:35346)
0806 02:57:06.049152 (+ 15397us) negotiation.cc:107] Waiting for socket to connect
0806 02:57:06.049175 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 02:57:06.049316 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:57:06.053692 (+ 4376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:57:06.053697 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:57:06.053720 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:57:06.054100 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:06.054108 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:06.055791 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:06.055796 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:06.057751 (+ 1955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:06.057768 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:06.059976 (+ 2208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:06.059983 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:06.060196 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:06.060229 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 02:57:06.060326 (+ 97us) client_negotiation.cc:240] Negotiation successful
0806 02:57:06.060459 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15222,"thread_start_us":15054,"threads_started":1}
I20260806 02:57:06.060935 21080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:06.034487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35346 (local address 127.25.211.131:37393)
0806 02:57:06.052880 (+ 18393us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:06.052888 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:06.052924 (+ 36us) server_negotiation.cc:407] Connection header received
0806 02:57:06.053014 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:06.053022 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:06.053268 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:06.053458 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:57:06.054319 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:06.055615 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:06.057995 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:06.059433 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:06.059628 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:06.059834 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:06.060395 (+ 561us) server_negotiation.cc:299] Negotiation successful
0806 02:57:06.060575 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18198,"thread_start_us":18039,"threads_started":1}
I20260806 02:57:06.064251 20953 tablet_service.cc:1467] Processing CreateTablet for tablet ade00a54ddee4b99add0819605268d7b (DEFAULT_TABLE table=test-table [id=08bd0aa1a4f944f4bd985b79d135171f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:57:06.064396 20760 tablet_service.cc:1467] Processing CreateTablet for tablet ade00a54ddee4b99add0819605268d7b (DEFAULT_TABLE table=test-table [id=08bd0aa1a4f944f4bd985b79d135171f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:57:06.065641 20953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ade00a54ddee4b99add0819605268d7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:57:06.065829 20760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ade00a54ddee4b99add0819605268d7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:57:06.067718 21076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:06.034111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:38215 (local address 127.0.0.1:45852)
0806 02:57:06.058370 (+ 24259us) negotiation.cc:107] Waiting for socket to connect
0806 02:57:06.058391 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 02:57:06.058625 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:57:06.059599 (+ 974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:57:06.059604 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:57:06.059632 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:57:06.060079 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:06.060093 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:06.062930 (+ 2837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:06.062935 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:06.064883 (+ 1948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:06.064901 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:06.066925 (+ 2024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:06.066933 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:06.067114 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:06.067148 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 02:57:06.067253 (+ 105us) client_negotiation.cc:240] Negotiation successful
0806 02:57:06.067397 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24127,"thread_start_us":23967,"threads_started":1}
I20260806 02:57:06.067958 21081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:06.035154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45852 (local address 127.25.211.129:38215)
0806 02:57:06.052263 (+ 17109us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:06.052272 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:06.058502 (+ 6230us) server_negotiation.cc:407] Connection header received
0806 02:57:06.058846 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:06.058854 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:06.059107 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:06.059291 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:57:06.060298 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:06.062778 (+ 2480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:06.065128 (+ 2350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:06.066698 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:06.066925 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:06.067163 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:06.067304 (+ 141us) server_negotiation.cc:299] Negotiation successful
0806 02:57:06.067494 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16911,"thread_start_us":16779,"threads_started":1}
I20260806 02:57:06.076498 20553 tablet_service.cc:1467] Processing CreateTablet for tablet ade00a54ddee4b99add0819605268d7b (DEFAULT_TABLE table=test-table [id=08bd0aa1a4f944f4bd985b79d135171f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:57:06.078065 20553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ade00a54ddee4b99add0819605268d7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:57:06.080952 21083 tablet_bootstrap.cc:492] T ade00a54ddee4b99add0819605268d7b P 333b1585ca0a47f2b268452c6396af54: Bootstrap starting.
I20260806 02:57:06.084277 21083 tablet_bootstrap.cc:654] T ade00a54ddee4b99add0819605268d7b P 333b1585ca0a47f2b268452c6396af54: Neither blocks nor log segments found. Creating new log.
I20260806 02:57:06.085362 21083 log.cc:824] T ade00a54ddee4b99add0819605268d7b P 333b1585ca0a47f2b268452c6396af54: Log is configured to *not* fsync() on all Append() calls
I20260806 02:57:06.089540 21086 tablet_bootstrap.cc:492] T ade00a54ddee4b99add0819605268d7b P 5803b20899584cf48dc1679fcaff801d: Bootstrap starting.
I20260806 02:57:06.090512 21083 tablet_bootstrap.cc:492] T ade00a54ddee4b99add0819605268d7b P 333b1585ca0a47f2b268452c6396af54: No bootstrap required, opened a new log
I20260806 02:57:06.090938 21083 ts_tablet_manager.cc:1397] T ade00a54ddee4b99add0819605268d7b P 333b1585ca0a47f2b268452c6396af54: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.004s
I20260806 02:57:06.093524 21086 tablet_bootstrap.cc:654] T ade00a54ddee4b99add0819605268d7b P 5803b20899584cf48dc1679fcaff801d: Neither blocks nor log segments found. Creating new log.
I20260806 02:57:06.094938 21086 log.cc:824] T ade00a54ddee4b99add0819605268d7b P 5803b20899584cf48dc1679fcaff801d: Log is configured to *not* fsync() on all Append() calls
I20260806 02:57:06.096576 21087 tablet_bootstrap.cc:492] T ade00a54ddee4b99add0819605268d7b P 0548a692f63442538f70b686f25beba6: Bootstrap starting.
I20260806 02:57:06.098332 21086 tablet_bootstrap.cc:492] T ade00a54ddee4b99add0819605268d7b P 5803b20899584cf48dc1679fcaff801d: No bootstrap required, opened a new log
I20260806 02:57:06.098649 21086 ts_tablet_manager.cc:1397] T ade00a54ddee4b99add0819605268d7b P 5803b20899584cf48dc1679fcaff801d: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260806 02:57:06.100579 21087 tablet_bootstrap.cc:654] T ade00a54ddee4b99add0819605268d7b P 0548a692f63442538f70b686f25beba6: Neither blocks nor log segments found. Creating new log.
I20260806 02:57:06.101914 21087 log.cc:824] T ade00a54ddee4b99add0819605268d7b P 0548a692f63442538f70b686f25beba6: Log is configured to *not* fsync() on all Append() calls
I20260806 02:57:06.102041 21083 raft_consensus.cc:348] T ade00a54ddee4b99add0819605268d7b P 333b1585ca0a47f2b268452c6396af54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333b1585ca0a47f2b268452c6396af54" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 38591 } } peers { permanent_uuid: "5803b20899584cf48dc1679fcaff801d" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 37393 } } peers { permanent_uuid: "0548a692f63442538f70b686f25beba6" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 38215 } }
I20260806 02:57:06.102587 21083 raft_consensus.cc:374] T ade00a54ddee4b99add0819605268d7b P 333b1585ca0a47f2b268452c6396af54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:57:06.102766 21083 raft_consensus.cc:729] T ade00a54ddee4b99add0819605268d7b P 333b1585ca0a47f2b268452c6396af54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 333b1585ca0a47f2b268452c6396af54, State: Initialized, Role: FOLLOWER
I20260806 02:57:06.103497 21083 consensus_queue.cc:260] T ade00a54ddee4b99add0819605268d7b P 333b1585ca0a47f2b268452c6396af54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333b1585ca0a47f2b268452c6396af54" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 38591 } } peers { permanent_uuid: "5803b20899584cf48dc1679fcaff801d" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 37393 } } peers { permanent_uuid: "0548a692f63442538f70b686f25beba6" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 38215 } }
I20260806 02:57:06.104672 20854 heartbeater.cc:503] Master 127.25.211.190:44373 was elected leader, sending a full tablet report...
I20260806 02:57:06.105576 21087 tablet_bootstrap.cc:492] T ade00a54ddee4b99add0819605268d7b P 0548a692f63442538f70b686f25beba6: No bootstrap required, opened a new log
I20260806 02:57:06.105934 21087 ts_tablet_manager.cc:1397] T ade00a54ddee4b99add0819605268d7b P 0548a692f63442538f70b686f25beba6: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260806 02:57:06.107379 21083 ts_tablet_manager.cc:1428] T ade00a54ddee4b99add0819605268d7b P 333b1585ca0a47f2b268452c6396af54: Time spent starting tablet: real 0.016s user 0.013s sys 0.000s
I20260806 02:57:06.108629 21086 raft_consensus.cc:348] T ade00a54ddee4b99add0819605268d7b P 5803b20899584cf48dc1679fcaff801d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333b1585ca0a47f2b268452c6396af54" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 38591 } } peers { permanent_uuid: "5803b20899584cf48dc1679fcaff801d" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 37393 } } peers { permanent_uuid: "0548a692f63442538f70b686f25beba6" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 38215 } }
I20260806 02:57:06.109143 21086 raft_consensus.cc:374] T ade00a54ddee4b99add0819605268d7b P 5803b20899584cf48dc1679fcaff801d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:57:06.109257 21086 raft_consensus.cc:729] T ade00a54ddee4b99add0819605268d7b P 5803b20899584cf48dc1679fcaff801d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5803b20899584cf48dc1679fcaff801d, State: Initialized, Role: FOLLOWER
I20260806 02:57:06.109949 21086 consensus_queue.cc:260] T ade00a54ddee4b99add0819605268d7b P 5803b20899584cf48dc1679fcaff801d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333b1585ca0a47f2b268452c6396af54" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 38591 } } peers { permanent_uuid: "5803b20899584cf48dc1679fcaff801d" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 37393 } } peers { permanent_uuid: "0548a692f63442538f70b686f25beba6" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 38215 } }
I20260806 02:57:06.111227 21051 heartbeater.cc:503] Master 127.25.211.190:44373 was elected leader, sending a full tablet report...
I20260806 02:57:06.111982 21086 ts_tablet_manager.cc:1428] T ade00a54ddee4b99add0819605268d7b P 5803b20899584cf48dc1679fcaff801d: Time spent starting tablet: real 0.013s user 0.005s sys 0.007s
I20260806 02:57:06.116398 21087 raft_consensus.cc:348] T ade00a54ddee4b99add0819605268d7b P 0548a692f63442538f70b686f25beba6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333b1585ca0a47f2b268452c6396af54" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 38591 } } peers { permanent_uuid: "5803b20899584cf48dc1679fcaff801d" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 37393 } } peers { permanent_uuid: "0548a692f63442538f70b686f25beba6" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 38215 } }
I20260806 02:57:06.116894 21087 raft_consensus.cc:374] T ade00a54ddee4b99add0819605268d7b P 0548a692f63442538f70b686f25beba6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:57:06.117000 21087 raft_consensus.cc:729] T ade00a54ddee4b99add0819605268d7b P 0548a692f63442538f70b686f25beba6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0548a692f63442538f70b686f25beba6, State: Initialized, Role: FOLLOWER
I20260806 02:57:06.117624 21087 consensus_queue.cc:260] T ade00a54ddee4b99add0819605268d7b P 0548a692f63442538f70b686f25beba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333b1585ca0a47f2b268452c6396af54" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 38591 } } peers { permanent_uuid: "5803b20899584cf48dc1679fcaff801d" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 37393 } } peers { permanent_uuid: "0548a692f63442538f70b686f25beba6" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 38215 } }
I20260806 02:57:06.119234 21087 ts_tablet_manager.cc:1428] T ade00a54ddee4b99add0819605268d7b P 0548a692f63442538f70b686f25beba6: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260806 02:57:06.188700 21092 raft_consensus.cc:482] T ade00a54ddee4b99add0819605268d7b P 333b1585ca0a47f2b268452c6396af54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:57:06.189085 21092 raft_consensus.cc:504] T ade00a54ddee4b99add0819605268d7b P 333b1585ca0a47f2b268452c6396af54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333b1585ca0a47f2b268452c6396af54" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 38591 } } peers { permanent_uuid: "5803b20899584cf48dc1679fcaff801d" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 37393 } } peers { permanent_uuid: "0548a692f63442538f70b686f25beba6" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 38215 } }
I20260806 02:57:06.190900 21092 leader_election.cc:288] T ade00a54ddee4b99add0819605268d7b P 333b1585ca0a47f2b268452c6396af54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5803b20899584cf48dc1679fcaff801d (127.25.211.131:37393), 0548a692f63442538f70b686f25beba6 (127.25.211.129:38215)
I20260806 02:57:06.200052 21080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:06.191362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:32785 (local address 127.25.211.131:37393)
0806 02:57:06.191599 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:06.191606 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:06.192354 (+ 748us) server_negotiation.cc:407] Connection header received
0806 02:57:06.192528 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:06.192534 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:06.192664 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:06.192749 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:57:06.193527 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:06.194661 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:06.197615 (+ 2954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:06.198907 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:06.199084 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:06.199158 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:06.199647 (+ 489us) server_negotiation.cc:299] Negotiation successful
0806 02:57:06.199791 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260806 02:57:06.200052 21081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:06.191505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:52237 (local address 127.25.211.129:38215)
0806 02:57:06.191695 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:06.191701 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:06.192354 (+ 653us) server_negotiation.cc:407] Connection header received
0806 02:57:06.192532 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:06.192537 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:06.192638 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:06.192749 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:57:06.193521 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:06.194660 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:06.197616 (+ 2956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:06.199001 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:06.199135 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:06.199195 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:06.199641 (+ 446us) server_negotiation.cc:299] Negotiation successful
0806 02:57:06.199812 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 02:57:06.200295 21098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:06.191389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:38215 (local address 127.25.211.130:52237)
0806 02:57:06.192214 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0806 02:57:06.192227 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:57:06.192411 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:57:06.192776 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:57:06.192781 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:57:06.192824 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:57:06.193295 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:06.193311 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:06.194793 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:06.194797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:06.197405 (+ 2608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:06.197415 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:06.199184 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:06.199189 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:06.199417 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:06.199526 (+ 109us) client_negotiation.cc:769] Sending connection context
0806 02:57:06.199612 (+ 86us) client_negotiation.cc:240] Negotiation successful
0806 02:57:06.199881 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"mutex_wait_us":230,"spinlock_wait_cycles":55424,"thread_start_us":388,"threads_started":1}
I20260806 02:57:06.200630 21100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:06.191295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.131:37393 (local address 127.25.211.130:32785)
0806 02:57:06.192184 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0806 02:57:06.192207 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 02:57:06.192378 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:57:06.192776 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:57:06.192781 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:57:06.192811 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:57:06.193371 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:06.193381 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:06.194789 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:06.194795 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:06.197389 (+ 2594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:06.197406 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:06.199111 (+ 1705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:06.199119 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:06.199422 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:06.199513 (+ 91us) client_negotiation.cc:769] Sending connection context
0806 02:57:06.199615 (+ 102us) client_negotiation.cc:240] Negotiation successful
0806 02:57:06.199901 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":712,"mutex_wait_us":281,"thread_start_us":567,"threads_started":1}
I20260806 02:57:06.201339 20584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ade00a54ddee4b99add0819605268d7b" candidate_uuid: "333b1585ca0a47f2b268452c6396af54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0548a692f63442538f70b686f25beba6" is_pre_election: true
I20260806 02:57:06.201617 20985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ade00a54ddee4b99add0819605268d7b" candidate_uuid: "333b1585ca0a47f2b268452c6396af54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5803b20899584cf48dc1679fcaff801d" is_pre_election: true
I20260806 02:57:06.201866 20584 raft_consensus.cc:2454] T ade00a54ddee4b99add0819605268d7b P 0548a692f63442538f70b686f25beba6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 333b1585ca0a47f2b268452c6396af54 in term 0.
I20260806 02:57:06.202175 20985 raft_consensus.cc:2454] T ade00a54ddee4b99add0819605268d7b P 5803b20899584cf48dc1679fcaff801d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 333b1585ca0a47f2b268452c6396af54 in term 0.
I20260806 02:57:06.202570 20690 leader_election.cc:302] T ade00a54ddee4b99add0819605268d7b P 333b1585ca0a47f2b268452c6396af54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0548a692f63442538f70b686f25beba6, 333b1585ca0a47f2b268452c6396af54; no voters:
I20260806 02:57:06.202997 21092 raft_consensus.cc:2781] T ade00a54ddee4b99add0819605268d7b P 333b1585ca0a47f2b268452c6396af54 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:57:06.203135 21092 raft_consensus.cc:482] T ade00a54ddee4b99add0819605268d7b P 333b1585ca0a47f2b268452c6396af54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:57:06.203243 21092 raft_consensus.cc:3037] T ade00a54ddee4b99add0819605268d7b P 333b1585ca0a47f2b268452c6396af54 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:57:06.205559 21092 raft_consensus.cc:504] T ade00a54ddee4b99add0819605268d7b P 333b1585ca0a47f2b268452c6396af54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333b1585ca0a47f2b268452c6396af54" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 38591 } } peers { permanent_uuid: "5803b20899584cf48dc1679fcaff801d" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 37393 } } peers { permanent_uuid: "0548a692f63442538f70b686f25beba6" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 38215 } }
I20260806 02:57:06.206203 21092 leader_election.cc:288] T ade00a54ddee4b99add0819605268d7b P 333b1585ca0a47f2b268452c6396af54 [CANDIDATE]: Term 1 election: Requested vote from peers 5803b20899584cf48dc1679fcaff801d (127.25.211.131:37393), 0548a692f63442538f70b686f25beba6 (127.25.211.129:38215)
I20260806 02:57:06.206629 20985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ade00a54ddee4b99add0819605268d7b" candidate_uuid: "333b1585ca0a47f2b268452c6396af54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5803b20899584cf48dc1679fcaff801d"
I20260806 02:57:06.206665 20584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ade00a54ddee4b99add0819605268d7b" candidate_uuid: "333b1585ca0a47f2b268452c6396af54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0548a692f63442538f70b686f25beba6"
I20260806 02:57:06.206854 20584 raft_consensus.cc:3037] T ade00a54ddee4b99add0819605268d7b P 0548a692f63442538f70b686f25beba6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:57:06.206854 20985 raft_consensus.cc:3037] T ade00a54ddee4b99add0819605268d7b P 5803b20899584cf48dc1679fcaff801d [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:57:06.210135 20584 raft_consensus.cc:2454] T ade00a54ddee4b99add0819605268d7b P 0548a692f63442538f70b686f25beba6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 333b1585ca0a47f2b268452c6396af54 in term 1.
I20260806 02:57:06.210134 20985 raft_consensus.cc:2454] T ade00a54ddee4b99add0819605268d7b P 5803b20899584cf48dc1679fcaff801d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 333b1585ca0a47f2b268452c6396af54 in term 1.
I20260806 02:57:06.210595 20681 leader_election.cc:302] T ade00a54ddee4b99add0819605268d7b P 333b1585ca0a47f2b268452c6396af54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 333b1585ca0a47f2b268452c6396af54, 5803b20899584cf48dc1679fcaff801d; no voters:
I20260806 02:57:06.210966 21092 raft_consensus.cc:2781] T ade00a54ddee4b99add0819605268d7b P 333b1585ca0a47f2b268452c6396af54 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:57:06.211436 21092 raft_consensus.cc:686] T ade00a54ddee4b99add0819605268d7b P 333b1585ca0a47f2b268452c6396af54 [term 1 LEADER]: Becoming Leader. State: Replica: 333b1585ca0a47f2b268452c6396af54, State: Running, Role: LEADER
I20260806 02:57:06.211915 21092 consensus_queue.cc:237] T ade00a54ddee4b99add0819605268d7b P 333b1585ca0a47f2b268452c6396af54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333b1585ca0a47f2b268452c6396af54" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 38591 } } peers { permanent_uuid: "5803b20899584cf48dc1679fcaff801d" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 37393 } } peers { permanent_uuid: "0548a692f63442538f70b686f25beba6" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 38215 } }
I20260806 02:57:06.216882 20376 catalog_manager.cc:5495] T ade00a54ddee4b99add0819605268d7b P 333b1585ca0a47f2b268452c6396af54 reported cstate change: term changed from 0 to 1, leader changed from <none> to 333b1585ca0a47f2b268452c6396af54 (127.25.211.130). New cstate: current_term: 1 leader_uuid: "333b1585ca0a47f2b268452c6396af54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "333b1585ca0a47f2b268452c6396af54" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 38591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5803b20899584cf48dc1679fcaff801d" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 37393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0548a692f63442538f70b686f25beba6" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 38215 } health_report { overall_health: UNKNOWN } } }
W20260806 02:57:06.224702 21053 tablet.cc:2369] T ade00a54ddee4b99add0819605268d7b P 5803b20899584cf48dc1679fcaff801d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:57:06.242478 20855 tablet.cc:2369] T ade00a54ddee4b99add0819605268d7b P 333b1585ca0a47f2b268452c6396af54: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:57:06.245313 21072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:06.239945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60880 (local address 127.25.211.130:38591)
0806 02:57:06.240152 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:06.240157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:06.240176 (+ 19us) server_negotiation.cc:407] Connection header received
0806 02:57:06.240269 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:06.240273 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:06.240363 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:06.240491 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:57:06.240865 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:06.241613 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:06.243081 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:06.243527 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:06.243650 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:06.244164 (+ 514us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:57:06.244692 (+ 528us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:57:06.244860 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:06.244957 (+ 97us) server_negotiation.cc:299] Negotiation successful
0806 02:57:06.245045 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 02:57:06.254029 20584 raft_consensus.cc:1261] T ade00a54ddee4b99add0819605268d7b P 0548a692f63442538f70b686f25beba6 [term 1 FOLLOWER]: Refusing update from remote peer 333b1585ca0a47f2b268452c6396af54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:57:06.254029 20985 raft_consensus.cc:1261] T ade00a54ddee4b99add0819605268d7b P 5803b20899584cf48dc1679fcaff801d [term 1 FOLLOWER]: Refusing update from remote peer 333b1585ca0a47f2b268452c6396af54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:57:06.255137 21101 consensus_queue.cc:1035] T ade00a54ddee4b99add0819605268d7b P 333b1585ca0a47f2b268452c6396af54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5803b20899584cf48dc1679fcaff801d" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 37393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:57:06.255482 21092 consensus_queue.cc:1035] T ade00a54ddee4b99add0819605268d7b P 333b1585ca0a47f2b268452c6396af54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0548a692f63442538f70b686f25beba6" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 38215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:57:06.268885 21106 mvcc.cc:205] Tried to move back new op lower bound from 7315394667525947392 to 7315394667367878656. Current Snapshot: MvccSnapshot[applied={T|T < 7315394667525947392}]
I20260806 02:57:06.270074 21107 mvcc.cc:205] Tried to move back new op lower bound from 7315394667525947392 to 7315394667367878656. Current Snapshot: MvccSnapshot[applied={T|T < 7315394667525947392}]
I20260806 02:57:06.282711 20376 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36772:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:57:06.282967 20376 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36772:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:57:06.286928 20376 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5d42ef1ebb93421fad1f9fb37499410f: Sending DeleteTablet for 3 replicas of tablet ade00a54ddee4b99add0819605268d7b
I20260806 02:57:06.288218 20760 tablet_service.cc:1514] Processing DeleteTablet for tablet ade00a54ddee4b99add0819605268d7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:57:06 UTC) from {username='slave'} at 127.0.0.1:60868
I20260806 02:57:06.288208 20953 tablet_service.cc:1514] Processing DeleteTablet for tablet ade00a54ddee4b99add0819605268d7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:57:06 UTC) from {username='slave'} at 127.0.0.1:35346
I20260806 02:57:06.288365 20553 tablet_service.cc:1514] Processing DeleteTablet for tablet ade00a54ddee4b99add0819605268d7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:57:06 UTC) from {username='slave'} at 127.0.0.1:45852
I20260806 02:57:06.290581 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 20459
I20260806 02:57:06.303814 21120 tablet_replica.cc:330] T ade00a54ddee4b99add0819605268d7b P 0548a692f63442538f70b686f25beba6: stopping tablet replica
I20260806 02:57:06.304529 21120 raft_consensus.cc:2229] T ade00a54ddee4b99add0819605268d7b P 0548a692f63442538f70b686f25beba6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:57:06.305155 21120 raft_consensus.cc:2258] T ade00a54ddee4b99add0819605268d7b P 0548a692f63442538f70b686f25beba6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:57:06.307533 21120 ts_tablet_manager.cc:1905] T ade00a54ddee4b99add0819605268d7b P 0548a692f63442538f70b686f25beba6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:57:06.314954 21120 ts_tablet_manager.cc:1918] T ade00a54ddee4b99add0819605268d7b P 0548a692f63442538f70b686f25beba6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:57:06.315169 21120 log.cc:1196] T ade00a54ddee4b99add0819605268d7b P 0548a692f63442538f70b686f25beba6: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/ts-0/wal/wals/ade00a54ddee4b99add0819605268d7b
I20260806 02:57:06.315740 21120 ts_tablet_manager.cc:1939] T ade00a54ddee4b99add0819605268d7b P 0548a692f63442538f70b686f25beba6: Deleting consensus metadata
I20260806 02:57:06.317723 20354 catalog_manager.cc:4841] TS 0548a692f63442538f70b686f25beba6 (127.25.211.129:38215): tablet ade00a54ddee4b99add0819605268d7b (table test-table [id=08bd0aa1a4f944f4bd985b79d135171f]) successfully deleted
I20260806 02:57:06.318775 21119 tablet_replica.cc:330] T ade00a54ddee4b99add0819605268d7b P 333b1585ca0a47f2b268452c6396af54: stopping tablet replica
I20260806 02:57:06.319319 21119 raft_consensus.cc:2229] T ade00a54ddee4b99add0819605268d7b P 333b1585ca0a47f2b268452c6396af54 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:57:06.320192 21119 raft_consensus.cc:2258] T ade00a54ddee4b99add0819605268d7b P 333b1585ca0a47f2b268452c6396af54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:57:06.320834 21118 tablet_replica.cc:330] T ade00a54ddee4b99add0819605268d7b P 5803b20899584cf48dc1679fcaff801d: stopping tablet replica
I20260806 02:57:06.321331 21118 raft_consensus.cc:2229] T ade00a54ddee4b99add0819605268d7b P 5803b20899584cf48dc1679fcaff801d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:57:06.321866 21118 raft_consensus.cc:2258] T ade00a54ddee4b99add0819605268d7b P 5803b20899584cf48dc1679fcaff801d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:57:06.323496 21119 ts_tablet_manager.cc:1905] T ade00a54ddee4b99add0819605268d7b P 333b1585ca0a47f2b268452c6396af54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:57:06.323985 21118 ts_tablet_manager.cc:1905] T ade00a54ddee4b99add0819605268d7b P 5803b20899584cf48dc1679fcaff801d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:57:06.327785 21081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:06.292174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45866 (local address 127.25.211.129:38215)
0806 02:57:06.292372 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:06.292380 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:06.316488 (+ 24108us) server_negotiation.cc:407] Connection header received
0806 02:57:06.316993 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:06.316999 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:06.317108 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:06.317182 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:06.317859 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:06.318955 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:06.320770 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:06.321374 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:06.321503 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:06.325922 (+ 4419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:06.325937 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:06.325955 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:57:06.326786 (+ 831us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:06.326854 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:57:06.326884 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:06.326994 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:06.327268 (+ 274us) server_negotiation.cc:299] Negotiation successful
0806 02:57:06.327404 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 02:57:06.329265 20638 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:57:06.331943 21119 ts_tablet_manager.cc:1918] T ade00a54ddee4b99add0819605268d7b P 333b1585ca0a47f2b268452c6396af54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:57:06.332233 21119 log.cc:1196] T ade00a54ddee4b99add0819605268d7b P 333b1585ca0a47f2b268452c6396af54: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/ts-1/wal/wals/ade00a54ddee4b99add0819605268d7b
I20260806 02:57:06.332471 21118 ts_tablet_manager.cc:1918] T ade00a54ddee4b99add0819605268d7b P 5803b20899584cf48dc1679fcaff801d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:57:06.332715 21118 log.cc:1196] T ade00a54ddee4b99add0819605268d7b P 5803b20899584cf48dc1679fcaff801d: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1785984707115702-26446-0/minicluster-data/ts-2/wal/wals/ade00a54ddee4b99add0819605268d7b
I20260806 02:57:06.333119 21119 ts_tablet_manager.cc:1939] T ade00a54ddee4b99add0819605268d7b P 333b1585ca0a47f2b268452c6396af54: Deleting consensus metadata
I20260806 02:57:06.333292 21118 ts_tablet_manager.cc:1939] T ade00a54ddee4b99add0819605268d7b P 5803b20899584cf48dc1679fcaff801d: Deleting consensus metadata
I20260806 02:57:06.335269 20352 catalog_manager.cc:4841] TS 5803b20899584cf48dc1679fcaff801d (127.25.211.131:37393): tablet ade00a54ddee4b99add0819605268d7b (table test-table [id=08bd0aa1a4f944f4bd985b79d135171f]) successfully deleted
I20260806 02:57:06.335302 20358 catalog_manager.cc:4841] TS 333b1585ca0a47f2b268452c6396af54 (127.25.211.130:38591): tablet ade00a54ddee4b99add0819605268d7b (table test-table [id=08bd0aa1a4f944f4bd985b79d135171f]) successfully deleted
I20260806 02:57:06.444476 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 20459
I20260806 02:57:06.479583 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 20652
I20260806 02:57:06.487468 21072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:06.480324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60896 (local address 127.25.211.130:38591)
0806 02:57:06.480519 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:06.480523 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:06.480582 (+ 59us) server_negotiation.cc:407] Connection header received
0806 02:57:06.480773 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:06.480778 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:06.480859 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:06.480936 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:06.481401 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:06.482240 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:06.484174 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:06.484613 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:06.484705 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:06.485955 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:06.485964 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:06.485979 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:57:06.486660 (+ 681us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:06.486714 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:57:06.486735 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:06.486825 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:06.486983 (+ 158us) server_negotiation.cc:299] Negotiation successful
0806 02:57:06.487086 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 02:57:06.488188 20848 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:57:06.601419 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 20652
I20260806 02:57:06.624598 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 20857
I20260806 02:57:06.631850 21080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:06.625422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35362 (local address 127.25.211.131:37393)
0806 02:57:06.625638 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:06.625644 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:06.625665 (+ 21us) server_negotiation.cc:407] Connection header received
0806 02:57:06.625845 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:06.625850 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:06.625942 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:06.626033 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:06.626489 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:06.627229 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:06.628585 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:06.628987 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:06.629107 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:06.630409 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:06.630420 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:06.630438 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:57:06.631018 (+ 580us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:06.631109 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:57:06.631133 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:06.631245 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:06.631392 (+ 147us) server_negotiation.cc:299] Negotiation successful
0806 02:57:06.631519 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 02:57:06.632663 21044 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:57:06.722668 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 20857
I20260806 02:57:06.743101 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 20327
I20260806 02:57:06.744282 20418 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:57:06.854972 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 20327
2026-08-06T02:57:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2934 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:57:06 dist-test-slave-zj8q krb5kdc[21130](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:57:06 dist-test-slave-zj8q krb5kdc[21130](info): set up 2 sockets
Aug 06 02:57:06 dist-test-slave-zj8q krb5kdc[21130](info): commencing operation
krb5kdc: starting...
W20260806 02:57:08.091475 26446 mini_kdc.cc:121] Time spent starting KDC: real 1.188s user 0.000s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Aug 06 02:57:08 dist-test-slave-zj8q krb5kdc[21130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:57:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:57:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.211.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab.
Entry for principal kudu/127.25.211.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab.
I20260806 02:57:08.421959 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:33833
--webserver_interface=127.25.211.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:35229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.211.190:33833
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:57:08.778636 21146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:57:08.778990 21146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:57:08.779134 21146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:57:08.786859 21146 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:57:08.787001 21146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:57:08.787086 21146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:57:08.787165 21146 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:57:08.787236 21146 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:57:08.797758 21146 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:35229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.211.190:33833
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:33833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.211.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:57:08.799189 21146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:57:08.800563 21146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:57:08.813441 21152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:08.813378 21153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:08.813464 21157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:57:08.814504 21146 server_base.cc:1034] running on GCE node
I20260806 02:57:08.815481 21146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:57:08.817122 21146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:57:08.818351 21146 hybrid_clock.cc:648] HybridClock initialized: now 1785985028818303 us; error 40 us; skew 500 ppm
I20260806 02:57:08.818795 21146 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:57:08 dist-test-slave-zj8q krb5kdc[21130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985028, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:57:08.822291 21146 init.cc:377] Logged in from keytab as kudu/127.25.211.190@KRBTEST.COM (short username kudu)
I20260806 02:57:08.824085 21146 webserver.cc:460] Webserver started at http://127.25.211.190:46137/ using document root <none> and password file <none>
I20260806 02:57:08.824856 21146 fs_manager.cc:359] Metadata directory not provided
I20260806 02:57:08.824990 21146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:57:08.825317 21146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:57:08.828687 21146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/master-0/data/instance:
uuid: "950e0370b249408cb9ab5111ea96ce92"
format_stamp: "Formatted at 2026-08-06 02:57:08 on dist-test-slave-zj8q"
server_key: "e656d8aba479b3a65da4e1aa987c11f3"
server_key_iv: "8078473cbc40ac1779a879edb08d8411"
server_key_version: "encryptionkey@0"
I20260806 02:57:08.829573 21146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/master-0/wal/instance:
uuid: "950e0370b249408cb9ab5111ea96ce92"
format_stamp: "Formatted at 2026-08-06 02:57:08 on dist-test-slave-zj8q"
server_key: "e656d8aba479b3a65da4e1aa987c11f3"
server_key_iv: "8078473cbc40ac1779a879edb08d8411"
server_key_version: "encryptionkey@0"
I20260806 02:57:08.835429 21146 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260806 02:57:08.839300 21169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:08.840732 21146 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 02:57:08.840925 21146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/master-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/master-0/wal
uuid: "950e0370b249408cb9ab5111ea96ce92"
format_stamp: "Formatted at 2026-08-06 02:57:08 on dist-test-slave-zj8q"
server_key: "e656d8aba479b3a65da4e1aa987c11f3"
server_key_iv: "8078473cbc40ac1779a879edb08d8411"
server_key_version: "encryptionkey@0"
I20260806 02:57:08.841141 21146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:57:08.866271 21146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:57:08.870764 21146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:57:08.871058 21146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:57:08.889737 21146 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.190:33833
I20260806 02:57:08.889744 21242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.190:33833 every 8 connection(s)
I20260806 02:57:08.891247 21146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
I20260806 02:57:08.894687 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 21146
I20260806 02:57:08.894858 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/master-0/wal/instance
I20260806 02:57:08.895320 26446 external_mini_cluster.cc:1428] Setting key cc7cf2818e53998c778ecb80b2563bd9
I20260806 02:57:08.895412 21243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:57:08.910905 21243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 950e0370b249408cb9ab5111ea96ce92: Bootstrap starting.
Aug 06 02:57:08 dist-test-slave-zj8q krb5kdc[21130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:57:08.914391 21243 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 950e0370b249408cb9ab5111ea96ce92: Neither blocks nor log segments found. Creating new log.
I20260806 02:57:08.915560 21243 log.cc:824] T 00000000000000000000000000000000 P 950e0370b249408cb9ab5111ea96ce92: Log is configured to *not* fsync() on all Append() calls
I20260806 02:57:08.918603 21243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 950e0370b249408cb9ab5111ea96ce92: No bootstrap required, opened a new log
I20260806 02:57:08.920209 21250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:08.897232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37612 (local address 127.25.211.190:33833)
0806 02:57:08.898069 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:08.898081 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:08.898129 (+ 48us) server_negotiation.cc:407] Connection header received
0806 02:57:08.900373 (+ 2244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:08.900399 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:08.900883 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:08.901244 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:08.903341 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:08.904754 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:08.906747 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:08.907503 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:08.907648 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:08.912746 (+ 5098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:08.912755 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:08.912771 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:57:08.912814 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:08.915944 (+ 3130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:08.916729 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:08.916737 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:08.916744 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:08.916924 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:08.917304 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:08.917308 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:08.917310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:08.917864 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:08.918222 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:08.918704 (+ 482us) server_negotiation.cc:299] Negotiation successful
0806 02:57:08.919169 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":383,"threads_started":1}
I20260806 02:57:08.926870 21243 raft_consensus.cc:348] T 00000000000000000000000000000000 P 950e0370b249408cb9ab5111ea96ce92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "950e0370b249408cb9ab5111ea96ce92" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 33833 } }
I20260806 02:57:08.927285 21243 raft_consensus.cc:374] T 00000000000000000000000000000000 P 950e0370b249408cb9ab5111ea96ce92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:57:08.927388 21243 raft_consensus.cc:729] T 00000000000000000000000000000000 P 950e0370b249408cb9ab5111ea96ce92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 950e0370b249408cb9ab5111ea96ce92, State: Initialized, Role: FOLLOWER
I20260806 02:57:08.927945 21243 consensus_queue.cc:260] T 00000000000000000000000000000000 P 950e0370b249408cb9ab5111ea96ce92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "950e0370b249408cb9ab5111ea96ce92" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 33833 } }
I20260806 02:57:08.928181 21243 raft_consensus.cc:388] T 00000000000000000000000000000000 P 950e0370b249408cb9ab5111ea96ce92 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:57:08.928313 21243 raft_consensus.cc:482] T 00000000000000000000000000000000 P 950e0370b249408cb9ab5111ea96ce92 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:57:08.928465 21243 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 950e0370b249408cb9ab5111ea96ce92 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:57:08.930806 21243 raft_consensus.cc:504] T 00000000000000000000000000000000 P 950e0370b249408cb9ab5111ea96ce92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "950e0370b249408cb9ab5111ea96ce92" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 33833 } }
I20260806 02:57:08.931332 21243 leader_election.cc:302] T 00000000000000000000000000000000 P 950e0370b249408cb9ab5111ea96ce92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 950e0370b249408cb9ab5111ea96ce92; no voters:
I20260806 02:57:08.931927 21243 leader_election.cc:288] T 00000000000000000000000000000000 P 950e0370b249408cb9ab5111ea96ce92 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:57:08.932065 21254 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 950e0370b249408cb9ab5111ea96ce92 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:57:08.932715 21254 raft_consensus.cc:686] T 00000000000000000000000000000000 P 950e0370b249408cb9ab5111ea96ce92 [term 1 LEADER]: Becoming Leader. State: Replica: 950e0370b249408cb9ab5111ea96ce92, State: Running, Role: LEADER
I20260806 02:57:08.933337 21254 consensus_queue.cc:237] T 00000000000000000000000000000000 P 950e0370b249408cb9ab5111ea96ce92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "950e0370b249408cb9ab5111ea96ce92" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 33833 } }
I20260806 02:57:08.934005 21243 sys_catalog.cc:564] T 00000000000000000000000000000000 P 950e0370b249408cb9ab5111ea96ce92 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:57:08.935928 21256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 950e0370b249408cb9ab5111ea96ce92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "950e0370b249408cb9ab5111ea96ce92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "950e0370b249408cb9ab5111ea96ce92" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 33833 } } }
I20260806 02:57:08.936071 21257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 950e0370b249408cb9ab5111ea96ce92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 950e0370b249408cb9ab5111ea96ce92. Latest consensus state: current_term: 1 leader_uuid: "950e0370b249408cb9ab5111ea96ce92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "950e0370b249408cb9ab5111ea96ce92" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 33833 } } }
I20260806 02:57:08.936419 21256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 950e0370b249408cb9ab5111ea96ce92 [sys.catalog]: This master's current role is: LEADER
I20260806 02:57:08.936427 21257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 950e0370b249408cb9ab5111ea96ce92 [sys.catalog]: This master's current role is: LEADER
I20260806 02:57:08.939374 21264 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:57:08.943892 21264 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:57:08.952561 21264 catalog_manager.cc:1347] Generated new cluster ID: c823ca281e664a0ca9e6e1741eed60eb
I20260806 02:57:08.952688 21264 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:57:08.972555 21264 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:57:08.973901 21264 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:57:08.985939 21264 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 950e0370b249408cb9ab5111ea96ce92: Generated new TSK 0
I20260806 02:57:08.987238 21264 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.211.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab.
Entry for principal kudu/127.25.211.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab.
I20260806 02:57:09.070116 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.129:0
--local_ip_for_outbound_sockets=127.25.211.129
--webserver_interface=127.25.211.129
--webserver_port=0
--tserver_master_addrs=127.25.211.190:33833
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab
--principal=kudu/127.25.211.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:35229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:57:09.420499 21283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:57:09.420820 21283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:57:09.420969 21283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:57:09.428440 21283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:57:09.428571 21283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:57:09.428725 21283 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.129
I20260806 02:57:09.439352 21283 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:35229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab
--principal=kudu/127.25.211.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.211.129
--webserver_port=0
--tserver_master_addrs=127.25.211.190:33833
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.129
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:57:09.440704 21283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:57:09.442061 21283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:57:09.454865 21293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:09.454928 21291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:09.454955 21296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:57:09.457119 21283 server_base.cc:1034] running on GCE node
I20260806 02:57:09.457758 21283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:57:09.458747 21283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:57:09.459972 21283 hybrid_clock.cc:648] HybridClock initialized: now 1785985029459913 us; error 45 us; skew 500 ppm
I20260806 02:57:09.460330 21283 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:57:09 dist-test-slave-zj8q krb5kdc[21130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985029, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:57:09.463519 21283 init.cc:377] Logged in from keytab as kudu/127.25.211.129@KRBTEST.COM (short username kudu)
I20260806 02:57:09.464951 21283 webserver.cc:460] Webserver started at http://127.25.211.129:37163/ using document root <none> and password file <none>
I20260806 02:57:09.465657 21283 fs_manager.cc:359] Metadata directory not provided
I20260806 02:57:09.465781 21283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:57:09.466084 21283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:57:09.468976 21283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/ts-0/data/instance:
uuid: "f949feba7e274b1ca2eda42b5f7ee930"
format_stamp: "Formatted at 2026-08-06 02:57:09 on dist-test-slave-zj8q"
server_key: "f89f8f29466b46ac6e4556ef0ed9cf5d"
server_key_iv: "5cb8c146e0a403301f1a7dce641135b2"
server_key_version: "encryptionkey@0"
I20260806 02:57:09.469712 21283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/ts-0/wal/instance:
uuid: "f949feba7e274b1ca2eda42b5f7ee930"
format_stamp: "Formatted at 2026-08-06 02:57:09 on dist-test-slave-zj8q"
server_key: "f89f8f29466b46ac6e4556ef0ed9cf5d"
server_key_iv: "5cb8c146e0a403301f1a7dce641135b2"
server_key_version: "encryptionkey@0"
I20260806 02:57:09.474756 21283 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260806 02:57:09.478252 21305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:09.479586 21283 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:57:09.479764 21283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/ts-0/wal
uuid: "f949feba7e274b1ca2eda42b5f7ee930"
format_stamp: "Formatted at 2026-08-06 02:57:09 on dist-test-slave-zj8q"
server_key: "f89f8f29466b46ac6e4556ef0ed9cf5d"
server_key_iv: "5cb8c146e0a403301f1a7dce641135b2"
server_key_version: "encryptionkey@0"
I20260806 02:57:09.479957 21283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:57:09.501616 21283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:57:09.505836 21283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:57:09.506109 21283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:57:09.507130 21283 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:57:09.508441 21283 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:57:09.508524 21283 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:09.508656 21283 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:57:09.508713 21283 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:09.538911 21283 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.129:45187
I20260806 02:57:09.538978 21463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.129:45187 every 8 connection(s)
I20260806 02:57:09.540199 21283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
I20260806 02:57:09.542979 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 21283
I20260806 02:57:09.543185 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/ts-0/wal/instance
I20260806 02:57:09.543620 26446 external_mini_cluster.cc:1428] Setting key d2b5a5036c416c86446f7cc524f3e577
Aug 06 02:57:09 dist-test-slave-zj8q krb5kdc[21130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985029, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.129@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:57:09.560889 21466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:09.542699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:32947 (local address 127.25.211.190:33833)
0806 02:57:09.543245 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:09.543252 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:09.544313 (+ 1061us) server_negotiation.cc:407] Connection header received
0806 02:57:09.546350 (+ 2037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:09.546355 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:09.546432 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:09.546525 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:09.548075 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:09.549051 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:09.550650 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:09.551172 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:09.551295 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:09.554878 (+ 3583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:09.554883 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:09.554887 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:57:09.554922 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:09.557473 (+ 2551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:09.558461 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:09.558465 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:09.558468 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:09.558607 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:09.559065 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:09.559070 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:09.559072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:09.559357 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:09.559502 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:09.560268 (+ 766us) server_negotiation.cc:299] Negotiation successful
0806 02:57:09.560516 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":271,"threads_started":1}
I20260806 02:57:09.561836 21468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:09.543268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:33833 (local address 127.25.211.129:32947)
0806 02:57:09.544087 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0806 02:57:09.544203 (+ 116us) client_negotiation.cc:174] Beginning negotiation
0806 02:57:09.546047 (+ 1844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:57:09.546742 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:57:09.546750 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:57:09.547302 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:57:09.547893 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:09.547912 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:09.549194 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:09.549198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:09.550509 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:09.550522 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:09.551367 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:09.551372 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:09.551573 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:09.552218 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:57:09.552240 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:57:09.554637 (+ 2397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:57:09.557662 (+ 3025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:57:09.557672 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:57:09.557690 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:57:09.558330 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:57:09.558715 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:57:09.558721 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:57:09.558725 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:57:09.558926 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:57:09.559495 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:57:09.559504 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:57:09.559857 (+ 353us) client_negotiation.cc:769] Sending connection context
0806 02:57:09.560113 (+ 256us) client_negotiation.cc:240] Negotiation successful
0806 02:57:09.560669 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":306,"threads_started":1}
I20260806 02:57:09.563594 21464 heartbeater.cc:347] Connected to a master server at 127.25.211.190:33833
I20260806 02:57:09.563906 21464 heartbeater.cc:464] Registering TS with master...
I20260806 02:57:09.564680 21464 heartbeater.cc:511] Master 127.25.211.190:33833 requested a full tablet report, sending...
I20260806 02:57:09.567095 21197 ts_manager.cc:194] Registered new tserver with Master: f949feba7e274b1ca2eda42b5f7ee930 (127.25.211.129:45187)
I20260806 02:57:09.568815 21197 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.211.129@KRBTEST.COM'} at 127.25.211.129:32947
WARNING: no policy specified for kudu/127.25.211.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab.
Entry for principal kudu/127.25.211.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab.
I20260806 02:57:09.613775 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.130:0
--local_ip_for_outbound_sockets=127.25.211.130
--webserver_interface=127.25.211.130
--webserver_port=0
--tserver_master_addrs=127.25.211.190:33833
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab
--principal=kudu/127.25.211.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:35229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:57:09.966696 21474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:57:09.967026 21474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:57:09.967171 21474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:57:09.974622 21474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:57:09.974774 21474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:57:09.974947 21474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.130
I20260806 02:57:09.985466 21474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:35229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab
--principal=kudu/127.25.211.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.211.130
--webserver_port=0
--tserver_master_addrs=127.25.211.190:33833
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.130
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:57:09.987103 21474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:57:09.988483 21474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:57:10.002259 21480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:10.002360 21481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:10.002324 21484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:57:10.003362 21474 server_base.cc:1034] running on GCE node
I20260806 02:57:10.004068 21474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:57:10.005203 21474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:57:10.006414 21474 hybrid_clock.cc:648] HybridClock initialized: now 1785985030006364 us; error 40 us; skew 500 ppm
I20260806 02:57:10.006767 21474 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:57:10 dist-test-slave-zj8q krb5kdc[21130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985030, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:57:10.010115 21474 init.cc:377] Logged in from keytab as kudu/127.25.211.130@KRBTEST.COM (short username kudu)
I20260806 02:57:10.011741 21474 webserver.cc:460] Webserver started at http://127.25.211.130:39837/ using document root <none> and password file <none>
I20260806 02:57:10.012442 21474 fs_manager.cc:359] Metadata directory not provided
I20260806 02:57:10.012557 21474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:57:10.012919 21474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:57:10.017195 21474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/ts-1/data/instance:
uuid: "4c0c62cb384a4ec28aff24207d9deb3c"
format_stamp: "Formatted at 2026-08-06 02:57:10 on dist-test-slave-zj8q"
server_key: "a4c3cdbdff3e15bf3598781ec22c2233"
server_key_iv: "e19a88cdc2c0ecd0ef81c885fb75ea5c"
server_key_version: "encryptionkey@0"
I20260806 02:57:10.018275 21474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/ts-1/wal/instance:
uuid: "4c0c62cb384a4ec28aff24207d9deb3c"
format_stamp: "Formatted at 2026-08-06 02:57:10 on dist-test-slave-zj8q"
server_key: "a4c3cdbdff3e15bf3598781ec22c2233"
server_key_iv: "e19a88cdc2c0ecd0ef81c885fb75ea5c"
server_key_version: "encryptionkey@0"
I20260806 02:57:10.024127 21474 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 02:57:10.027954 21491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:10.029410 21474 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 02:57:10.029583 21474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/ts-1/wal
uuid: "4c0c62cb384a4ec28aff24207d9deb3c"
format_stamp: "Formatted at 2026-08-06 02:57:10 on dist-test-slave-zj8q"
server_key: "a4c3cdbdff3e15bf3598781ec22c2233"
server_key_iv: "e19a88cdc2c0ecd0ef81c885fb75ea5c"
server_key_version: "encryptionkey@0"
I20260806 02:57:10.029793 21474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:57:10.048197 21474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:57:10.052583 21474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:57:10.052884 21474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:57:10.053931 21474 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:57:10.055436 21474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:57:10.055521 21474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:10.055652 21474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:57:10.055706 21474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:10.088228 21474 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.130:37419
I20260806 02:57:10.088279 21660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.130:37419 every 8 connection(s)
I20260806 02:57:10.089578 21474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
I20260806 02:57:10.096053 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 21474
I20260806 02:57:10.096271 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/ts-1/wal/instance
I20260806 02:57:10.096843 26446 external_mini_cluster.cc:1428] Setting key 8ee9e797d5143f951fb25234e8060819
Aug 06 02:57:10 dist-test-slave-zj8q krb5kdc[21130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985030, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.130@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:57:10.111850 21665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:10.092286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:60991 (local address 127.25.211.190:33833)
0806 02:57:10.092734 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:10.092739 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:10.093840 (+ 1101us) server_negotiation.cc:407] Connection header received
0806 02:57:10.095924 (+ 2084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:10.095929 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:10.096007 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:10.096131 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:10.098319 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:10.099163 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:10.100759 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:10.101189 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:10.101300 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:10.105323 (+ 4023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:10.105329 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:10.105334 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:57:10.105373 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:10.108289 (+ 2916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:10.109216 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:10.109221 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:10.109224 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:10.109378 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:10.109855 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:10.109859 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:10.109862 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:10.110217 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:10.110344 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:10.111260 (+ 916us) server_negotiation.cc:299] Negotiation successful
0806 02:57:10.111488 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":223,"threads_started":1}
I20260806 02:57:10.113161 21670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:10.092811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:33833 (local address 127.25.211.130:60991)
0806 02:57:10.093644 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0806 02:57:10.093686 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 02:57:10.095491 (+ 1805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:57:10.096457 (+ 966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:57:10.096470 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:57:10.097306 (+ 836us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:57:10.098115 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:10.098139 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:10.099324 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:10.099327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:10.100613 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:10.100628 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:10.101433 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:10.101442 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:10.101677 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:10.102413 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:57:10.102444 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:57:10.105074 (+ 2630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:57:10.108454 (+ 3380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:57:10.108465 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:57:10.108484 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:57:10.109097 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:57:10.109482 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:57:10.109486 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:57:10.109490 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:57:10.109755 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:57:10.110351 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:57:10.110359 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:57:10.110829 (+ 470us) client_negotiation.cc:769] Sending connection context
0806 02:57:10.111160 (+ 331us) client_negotiation.cc:240] Negotiation successful
0806 02:57:10.111729 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":331,"threads_started":1}
I20260806 02:57:10.114797 21661 heartbeater.cc:347] Connected to a master server at 127.25.211.190:33833
I20260806 02:57:10.115176 21661 heartbeater.cc:464] Registering TS with master...
I20260806 02:57:10.116134 21661 heartbeater.cc:511] Master 127.25.211.190:33833 requested a full tablet report, sending...
I20260806 02:57:10.117882 21197 ts_manager.cc:194] Registered new tserver with Master: 4c0c62cb384a4ec28aff24207d9deb3c (127.25.211.130:37419)
I20260806 02:57:10.118743 21197 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.211.130@KRBTEST.COM'} at 127.25.211.130:60991
WARNING: no policy specified for kudu/127.25.211.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab.
Entry for principal kudu/127.25.211.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab.
I20260806 02:57:10.169237 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.131:0
--local_ip_for_outbound_sockets=127.25.211.131
--webserver_interface=127.25.211.131
--webserver_port=0
--tserver_master_addrs=127.25.211.190:33833
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab
--principal=kudu/127.25.211.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:35229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:57:10.508526 21674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:57:10.508858 21674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:57:10.509008 21674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:57:10.516372 21674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:57:10.516543 21674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:57:10.516764 21674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.131
I20260806 02:57:10.527179 21674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:35229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab
--principal=kudu/127.25.211.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.211.131
--webserver_port=0
--tserver_master_addrs=127.25.211.190:33833
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.131
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:57:10.528494 21674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:57:10.529846 21674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:57:10.542667 21679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:10.542706 21680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:10.543092 21682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:57:10.543970 21674 server_base.cc:1034] running on GCE node
I20260806 02:57:10.544662 21674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:57:10.545644 21674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:57:10.546869 21674 hybrid_clock.cc:648] HybridClock initialized: now 1785985030546801 us; error 45 us; skew 500 ppm
I20260806 02:57:10.547204 21674 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:57:10 dist-test-slave-zj8q krb5kdc[21130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985030, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:57:10.550509 21674 init.cc:377] Logged in from keytab as kudu/127.25.211.131@KRBTEST.COM (short username kudu)
I20260806 02:57:10.552155 21674 webserver.cc:460] Webserver started at http://127.25.211.131:42181/ using document root <none> and password file <none>
I20260806 02:57:10.552882 21674 fs_manager.cc:359] Metadata directory not provided
I20260806 02:57:10.553010 21674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:57:10.553325 21674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:57:10.556401 21674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/ts-2/data/instance:
uuid: "a60e56aca01147d2a3af980397d153c9"
format_stamp: "Formatted at 2026-08-06 02:57:10 on dist-test-slave-zj8q"
server_key: "d0d7ac4733d7fdc1e69b994531007e9b"
server_key_iv: "a0463d4c25f6d5d069d8a8a3f0fa9ec4"
server_key_version: "encryptionkey@0"
I20260806 02:57:10.557204 21674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/ts-2/wal/instance:
uuid: "a60e56aca01147d2a3af980397d153c9"
format_stamp: "Formatted at 2026-08-06 02:57:10 on dist-test-slave-zj8q"
server_key: "d0d7ac4733d7fdc1e69b994531007e9b"
server_key_iv: "a0463d4c25f6d5d069d8a8a3f0fa9ec4"
server_key_version: "encryptionkey@0"
I20260806 02:57:10.562770 21674 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260806 02:57:10.566849 21694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:10.568429 21674 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:57:10.568632 21674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/ts-2/wal
uuid: "a60e56aca01147d2a3af980397d153c9"
format_stamp: "Formatted at 2026-08-06 02:57:10 on dist-test-slave-zj8q"
server_key: "d0d7ac4733d7fdc1e69b994531007e9b"
server_key_iv: "a0463d4c25f6d5d069d8a8a3f0fa9ec4"
server_key_version: "encryptionkey@0"
I20260806 02:57:10.568871 21674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:57:10.572301 21464 heartbeater.cc:503] Master 127.25.211.190:33833 was elected leader, sending a full tablet report...
I20260806 02:57:10.588392 21674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:57:10.593111 21674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:57:10.593452 21674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:57:10.594751 21674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:57:10.596308 21674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:57:10.596396 21674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:10.596534 21674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:57:10.596599 21674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:10.628096 21674 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.131:37633
I20260806 02:57:10.628166 21859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.131:37633 every 8 connection(s)
I20260806 02:57:10.629379 21674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
I20260806 02:57:10.631295 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 21674
I20260806 02:57:10.631459 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/ts-2/wal/instance
I20260806 02:57:10.631948 26446 external_mini_cluster.cc:1428] Setting key fafd866d19fdd7ebccb1b36f1b2a54b1
Aug 06 02:57:10 dist-test-slave-zj8q krb5kdc[21130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985030, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.131@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:57:10.650311 21866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:10.632075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:38123 (local address 127.25.211.190:33833)
0806 02:57:10.632563 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:10.632570 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:10.633869 (+ 1299us) server_negotiation.cc:407] Connection header received
0806 02:57:10.635919 (+ 2050us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:10.635922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:10.635999 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:10.636109 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:10.637645 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:10.638503 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:10.640331 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:10.640753 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:10.640851 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:10.644301 (+ 3450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:10.644306 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:10.644310 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:57:10.644346 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:10.647401 (+ 3055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:10.648191 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:10.648195 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:10.648198 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:10.648343 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:10.648731 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:10.648735 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:10.648737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:10.649007 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:10.649130 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:10.649802 (+ 672us) server_negotiation.cc:299] Negotiation successful
0806 02:57:10.649989 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":209,"threads_started":1}
I20260806 02:57:10.651326 21867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:10.632680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:33833 (local address 127.25.211.131:38123)
0806 02:57:10.633554 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0806 02:57:10.633668 (+ 114us) client_negotiation.cc:174] Beginning negotiation
0806 02:57:10.635688 (+ 2020us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:57:10.636367 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:57:10.636375 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:57:10.636909 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:57:10.637466 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:10.637484 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:10.638665 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:10.638669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:10.640185 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:10.640200 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:10.640976 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:10.640982 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:10.641214 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:10.641858 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:57:10.641880 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:57:10.644079 (+ 2199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:57:10.647535 (+ 3456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:57:10.647546 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:57:10.647573 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:57:10.648072 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:57:10.648435 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:57:10.648438 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:57:10.648440 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:57:10.648648 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:57:10.649113 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:57:10.649120 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:57:10.649475 (+ 355us) client_negotiation.cc:769] Sending connection context
0806 02:57:10.649728 (+ 253us) client_negotiation.cc:240] Negotiation successful
0806 02:57:10.650192 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":336,"threads_started":1}
I20260806 02:57:10.652699 21861 heartbeater.cc:347] Connected to a master server at 127.25.211.190:33833
I20260806 02:57:10.652984 21861 heartbeater.cc:464] Registering TS with master...
I20260806 02:57:10.653678 21861 heartbeater.cc:511] Master 127.25.211.190:33833 requested a full tablet report, sending...
I20260806 02:57:10.655110 21197 ts_manager.cc:194] Registered new tserver with Master: a60e56aca01147d2a3af980397d153c9 (127.25.211.131:37633)
I20260806 02:57:10.655892 21197 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.211.131@KRBTEST.COM'} at 127.25.211.131:38123
I20260806 02:57:10.659735 26446 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:57:10.676959 21866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:10.663715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37628 (local address 127.25.211.190:33833)
0806 02:57:10.663894 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:10.663901 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:10.664256 (+ 355us) server_negotiation.cc:407] Connection header received
0806 02:57:10.664432 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:10.664436 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:10.664508 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:10.664589 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:10.666079 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:10.666859 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:10.668145 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:10.668561 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:10.668667 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:10.671382 (+ 2715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:10.671388 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:10.671391 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:57:10.671429 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:10.673963 (+ 2534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:10.674846 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:10.674851 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:10.674855 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:10.674990 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:10.675455 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:10.675461 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:10.675463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:10.675783 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:10.676043 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:10.676441 (+ 398us) server_negotiation.cc:299] Negotiation successful
0806 02:57:10.676663 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 02:57:10.680940 21197 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37628:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:57:10.687953 21197 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:57:10.725302 21884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:10.700334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46226 (local address 127.25.211.131:37633)
0806 02:57:10.716507 (+ 16173us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:10.716516 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:10.716555 (+ 39us) server_negotiation.cc:407] Connection header received
0806 02:57:10.716669 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:10.716679 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:10.716965 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:10.717184 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:57:10.718344 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:10.719991 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:10.722673 (+ 2682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:10.724112 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:10.724301 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:10.724496 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:10.724786 (+ 290us) server_negotiation.cc:299] Negotiation successful
0806 02:57:10.724948 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15993,"thread_start_us":15825,"threads_started":1}
I20260806 02:57:10.725603 21883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:10.700191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.131:37633 (local address 127.0.0.1:46226)
0806 02:57:10.715113 (+ 14922us) negotiation.cc:107] Waiting for socket to connect
0806 02:57:10.715194 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0806 02:57:10.715400 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:57:10.717558 (+ 2158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:57:10.717568 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:57:10.717605 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:57:10.718060 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:10.718074 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:10.720160 (+ 2086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:10.720165 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:10.722489 (+ 2324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:10.722505 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:10.724325 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:10.724331 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:10.724533 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:10.724583 (+ 50us) client_negotiation.cc:769] Sending connection context
0806 02:57:10.724701 (+ 118us) client_negotiation.cc:240] Negotiation successful
0806 02:57:10.724907 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14619,"mutex_wait_us":4,"thread_start_us":15429,"threads_started":1}
I20260806 02:57:10.728200 21885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:10.700601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.130:37419 (local address 127.0.0.1:46834)
0806 02:57:10.719664 (+ 19063us) negotiation.cc:107] Waiting for socket to connect
0806 02:57:10.719683 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:57:10.719814 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:57:10.720967 (+ 1153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:57:10.720971 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:57:10.720995 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:57:10.721810 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:10.721823 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:10.723377 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:10.723382 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:10.725680 (+ 2298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:10.725696 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:10.727381 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:10.727387 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:10.727561 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:10.727593 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 02:57:10.727684 (+ 91us) client_negotiation.cc:240] Negotiation successful
0806 02:57:10.727826 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18936,"mutex_wait_us":4,"thread_start_us":18784,"threads_started":1}
I20260806 02:57:10.728338 21887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:10.701057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46834 (local address 127.25.211.130:37419)
0806 02:57:10.719279 (+ 18222us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:10.719289 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:10.719975 (+ 686us) server_negotiation.cc:407] Connection header received
0806 02:57:10.720111 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:10.720119 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:10.720369 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:10.720580 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:57:10.722029 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:10.723241 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:10.725861 (+ 2620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:10.727170 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:10.727326 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:10.727493 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:10.727845 (+ 352us) server_negotiation.cc:299] Negotiation successful
0806 02:57:10.728010 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18043,"thread_start_us":17917,"threads_started":1}
I20260806 02:57:10.729698 21890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:10.719771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:45187 (local address 127.0.0.1:39110)
0806 02:57:10.720670 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0806 02:57:10.720689 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:57:10.720807 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:57:10.721706 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:57:10.721711 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:57:10.721734 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:57:10.722127 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:10.722144 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:10.723918 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:10.723923 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:10.726047 (+ 2124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:10.726060 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:10.728977 (+ 2917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:10.728984 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:10.729196 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:10.729240 (+ 44us) client_negotiation.cc:769] Sending connection context
0806 02:57:10.729335 (+ 95us) client_negotiation.cc:240] Negotiation successful
0806 02:57:10.729435 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":776,"mutex_wait_us":4,"thread_start_us":999,"threads_started":1}
I20260806 02:57:10.730372 21888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:10.716933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39110 (local address 127.25.211.129:45187)
0806 02:57:10.720365 (+ 3432us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:10.720374 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:10.720935 (+ 561us) server_negotiation.cc:407] Connection header received
0806 02:57:10.721048 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:10.721055 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:10.721272 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:10.721460 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:57:10.722315 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:10.723770 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:10.726218 (+ 2448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:10.728686 (+ 2468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:10.728884 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:10.729051 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:10.729973 (+ 922us) server_negotiation.cc:299] Negotiation successful
0806 02:57:10.730100 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3261,"thread_start_us":3104,"threads_started":1}
I20260806 02:57:10.730229 21765 tablet_service.cc:1467] Processing CreateTablet for tablet eb6c47e130d645b4aab75a9d8297709b (DEFAULT_TABLE table=test-table [id=b70afa7612f0498c8aaa642272986ca1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:57:10.732044 21571 tablet_service.cc:1467] Processing CreateTablet for tablet eb6c47e130d645b4aab75a9d8297709b (DEFAULT_TABLE table=test-table [id=b70afa7612f0498c8aaa642272986ca1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:57:10.732786 21765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb6c47e130d645b4aab75a9d8297709b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:57:10.733456 21571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb6c47e130d645b4aab75a9d8297709b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:57:10.733304 21372 tablet_service.cc:1467] Processing CreateTablet for tablet eb6c47e130d645b4aab75a9d8297709b (DEFAULT_TABLE table=test-table [id=b70afa7612f0498c8aaa642272986ca1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:57:10.734756 21372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb6c47e130d645b4aab75a9d8297709b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:57:10.747259 21892 tablet_bootstrap.cc:492] T eb6c47e130d645b4aab75a9d8297709b P f949feba7e274b1ca2eda42b5f7ee930: Bootstrap starting.
I20260806 02:57:10.748377 21894 tablet_bootstrap.cc:492] T eb6c47e130d645b4aab75a9d8297709b P a60e56aca01147d2a3af980397d153c9: Bootstrap starting.
I20260806 02:57:10.751474 21892 tablet_bootstrap.cc:654] T eb6c47e130d645b4aab75a9d8297709b P f949feba7e274b1ca2eda42b5f7ee930: Neither blocks nor log segments found. Creating new log.
I20260806 02:57:10.751986 21894 tablet_bootstrap.cc:654] T eb6c47e130d645b4aab75a9d8297709b P a60e56aca01147d2a3af980397d153c9: Neither blocks nor log segments found. Creating new log.
I20260806 02:57:10.752946 21892 log.cc:824] T eb6c47e130d645b4aab75a9d8297709b P f949feba7e274b1ca2eda42b5f7ee930: Log is configured to *not* fsync() on all Append() calls
I20260806 02:57:10.753239 21894 log.cc:824] T eb6c47e130d645b4aab75a9d8297709b P a60e56aca01147d2a3af980397d153c9: Log is configured to *not* fsync() on all Append() calls
I20260806 02:57:10.754544 21893 tablet_bootstrap.cc:492] T eb6c47e130d645b4aab75a9d8297709b P 4c0c62cb384a4ec28aff24207d9deb3c: Bootstrap starting.
I20260806 02:57:10.757323 21894 tablet_bootstrap.cc:492] T eb6c47e130d645b4aab75a9d8297709b P a60e56aca01147d2a3af980397d153c9: No bootstrap required, opened a new log
I20260806 02:57:10.757586 21892 tablet_bootstrap.cc:492] T eb6c47e130d645b4aab75a9d8297709b P f949feba7e274b1ca2eda42b5f7ee930: No bootstrap required, opened a new log
I20260806 02:57:10.757597 21894 ts_tablet_manager.cc:1397] T eb6c47e130d645b4aab75a9d8297709b P a60e56aca01147d2a3af980397d153c9: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260806 02:57:10.757858 21892 ts_tablet_manager.cc:1397] T eb6c47e130d645b4aab75a9d8297709b P f949feba7e274b1ca2eda42b5f7ee930: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260806 02:57:10.758298 21893 tablet_bootstrap.cc:654] T eb6c47e130d645b4aab75a9d8297709b P 4c0c62cb384a4ec28aff24207d9deb3c: Neither blocks nor log segments found. Creating new log.
I20260806 02:57:10.759575 21893 log.cc:824] T eb6c47e130d645b4aab75a9d8297709b P 4c0c62cb384a4ec28aff24207d9deb3c: Log is configured to *not* fsync() on all Append() calls
I20260806 02:57:10.763150 21893 tablet_bootstrap.cc:492] T eb6c47e130d645b4aab75a9d8297709b P 4c0c62cb384a4ec28aff24207d9deb3c: No bootstrap required, opened a new log
I20260806 02:57:10.763455 21893 ts_tablet_manager.cc:1397] T eb6c47e130d645b4aab75a9d8297709b P 4c0c62cb384a4ec28aff24207d9deb3c: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260806 02:57:10.765475 21894 raft_consensus.cc:348] T eb6c47e130d645b4aab75a9d8297709b P a60e56aca01147d2a3af980397d153c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a60e56aca01147d2a3af980397d153c9" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 37633 } } peers { permanent_uuid: "f949feba7e274b1ca2eda42b5f7ee930" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 45187 } } peers { permanent_uuid: "4c0c62cb384a4ec28aff24207d9deb3c" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 37419 } }
I20260806 02:57:10.765915 21894 raft_consensus.cc:374] T eb6c47e130d645b4aab75a9d8297709b P a60e56aca01147d2a3af980397d153c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:57:10.766027 21894 raft_consensus.cc:729] T eb6c47e130d645b4aab75a9d8297709b P a60e56aca01147d2a3af980397d153c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a60e56aca01147d2a3af980397d153c9, State: Initialized, Role: FOLLOWER
I20260806 02:57:10.766654 21894 consensus_queue.cc:260] T eb6c47e130d645b4aab75a9d8297709b P a60e56aca01147d2a3af980397d153c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a60e56aca01147d2a3af980397d153c9" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 37633 } } peers { permanent_uuid: "f949feba7e274b1ca2eda42b5f7ee930" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 45187 } } peers { permanent_uuid: "4c0c62cb384a4ec28aff24207d9deb3c" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 37419 } }
I20260806 02:57:10.767781 21861 heartbeater.cc:503] Master 127.25.211.190:33833 was elected leader, sending a full tablet report...
I20260806 02:57:10.768461 21894 ts_tablet_manager.cc:1428] T eb6c47e130d645b4aab75a9d8297709b P a60e56aca01147d2a3af980397d153c9: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260806 02:57:10.769754 21892 raft_consensus.cc:348] T eb6c47e130d645b4aab75a9d8297709b P f949feba7e274b1ca2eda42b5f7ee930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a60e56aca01147d2a3af980397d153c9" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 37633 } } peers { permanent_uuid: "f949feba7e274b1ca2eda42b5f7ee930" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 45187 } } peers { permanent_uuid: "4c0c62cb384a4ec28aff24207d9deb3c" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 37419 } }
I20260806 02:57:10.770226 21892 raft_consensus.cc:374] T eb6c47e130d645b4aab75a9d8297709b P f949feba7e274b1ca2eda42b5f7ee930 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:57:10.770327 21892 raft_consensus.cc:729] T eb6c47e130d645b4aab75a9d8297709b P f949feba7e274b1ca2eda42b5f7ee930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f949feba7e274b1ca2eda42b5f7ee930, State: Initialized, Role: FOLLOWER
I20260806 02:57:10.770937 21892 consensus_queue.cc:260] T eb6c47e130d645b4aab75a9d8297709b P f949feba7e274b1ca2eda42b5f7ee930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a60e56aca01147d2a3af980397d153c9" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 37633 } } peers { permanent_uuid: "f949feba7e274b1ca2eda42b5f7ee930" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 45187 } } peers { permanent_uuid: "4c0c62cb384a4ec28aff24207d9deb3c" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 37419 } }
I20260806 02:57:10.773808 21893 raft_consensus.cc:348] T eb6c47e130d645b4aab75a9d8297709b P 4c0c62cb384a4ec28aff24207d9deb3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a60e56aca01147d2a3af980397d153c9" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 37633 } } peers { permanent_uuid: "f949feba7e274b1ca2eda42b5f7ee930" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 45187 } } peers { permanent_uuid: "4c0c62cb384a4ec28aff24207d9deb3c" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 37419 } }
I20260806 02:57:10.774348 21893 raft_consensus.cc:374] T eb6c47e130d645b4aab75a9d8297709b P 4c0c62cb384a4ec28aff24207d9deb3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:57:10.774452 21893 raft_consensus.cc:729] T eb6c47e130d645b4aab75a9d8297709b P 4c0c62cb384a4ec28aff24207d9deb3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c0c62cb384a4ec28aff24207d9deb3c, State: Initialized, Role: FOLLOWER
I20260806 02:57:10.775168 21893 consensus_queue.cc:260] T eb6c47e130d645b4aab75a9d8297709b P 4c0c62cb384a4ec28aff24207d9deb3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a60e56aca01147d2a3af980397d153c9" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 37633 } } peers { permanent_uuid: "f949feba7e274b1ca2eda42b5f7ee930" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 45187 } } peers { permanent_uuid: "4c0c62cb384a4ec28aff24207d9deb3c" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 37419 } }
I20260806 02:57:10.775710 21892 ts_tablet_manager.cc:1428] T eb6c47e130d645b4aab75a9d8297709b P f949feba7e274b1ca2eda42b5f7ee930: Time spent starting tablet: real 0.018s user 0.009s sys 0.008s
I20260806 02:57:10.776436 21661 heartbeater.cc:503] Master 127.25.211.190:33833 was elected leader, sending a full tablet report...
I20260806 02:57:10.776933 21893 ts_tablet_manager.cc:1428] T eb6c47e130d645b4aab75a9d8297709b P 4c0c62cb384a4ec28aff24207d9deb3c: Time spent starting tablet: real 0.013s user 0.004s sys 0.009s
W20260806 02:57:10.792593 21465 tablet.cc:2369] T eb6c47e130d645b4aab75a9d8297709b P f949feba7e274b1ca2eda42b5f7ee930: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:57:10.841586 21663 tablet.cc:2369] T eb6c47e130d645b4aab75a9d8297709b P 4c0c62cb384a4ec28aff24207d9deb3c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:57:10.852118 21905 raft_consensus.cc:482] T eb6c47e130d645b4aab75a9d8297709b P a60e56aca01147d2a3af980397d153c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:57:10.852435 21905 raft_consensus.cc:504] T eb6c47e130d645b4aab75a9d8297709b P a60e56aca01147d2a3af980397d153c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a60e56aca01147d2a3af980397d153c9" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 37633 } } peers { permanent_uuid: "f949feba7e274b1ca2eda42b5f7ee930" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 45187 } } peers { permanent_uuid: "4c0c62cb384a4ec28aff24207d9deb3c" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 37419 } }
I20260806 02:57:10.854255 21905 leader_election.cc:288] T eb6c47e130d645b4aab75a9d8297709b P a60e56aca01147d2a3af980397d153c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f949feba7e274b1ca2eda42b5f7ee930 (127.25.211.129:45187), 4c0c62cb384a4ec28aff24207d9deb3c (127.25.211.130:37419)
I20260806 02:57:10.860770 21887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:10.854787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:44309 (local address 127.25.211.130:37419)
0806 02:57:10.854973 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:10.854978 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:10.854996 (+ 18us) server_negotiation.cc:407] Connection header received
0806 02:57:10.855265 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:10.855271 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:10.855355 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:10.855468 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:57:10.855871 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:10.856822 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:10.858842 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:10.859928 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:10.860054 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:10.860110 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:10.860428 (+ 318us) server_negotiation.cc:299] Negotiation successful
0806 02:57:10.860518 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 02:57:10.861054 21867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:10.854611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.130:37419 (local address 127.25.211.131:44309)
0806 02:57:10.854920 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0806 02:57:10.854942 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 02:57:10.855084 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:57:10.855494 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:57:10.855497 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:57:10.855528 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:57:10.855731 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:10.855743 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:10.856965 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:10.856971 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:10.858659 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:10.858675 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:10.860075 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:10.860082 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:10.860274 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:10.860312 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 02:57:10.860415 (+ 103us) client_negotiation.cc:240] Negotiation successful
0806 02:57:10.860569 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":3}
I20260806 02:57:10.861626 21888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:10.854719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:55179 (local address 127.25.211.129:45187)
0806 02:57:10.854920 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:10.854927 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:10.855619 (+ 692us) server_negotiation.cc:407] Connection header received
0806 02:57:10.855777 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:10.855782 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:10.855877 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:10.855946 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:57:10.856546 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:10.857463 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:10.859603 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:10.860707 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:10.860851 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:10.860937 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:10.861338 (+ 401us) server_negotiation.cc:299] Negotiation successful
0806 02:57:10.861454 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 02:57:10.861795 21908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:10.854724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:45187 (local address 127.25.211.131:55179)
0806 02:57:10.855499 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0806 02:57:10.855518 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:57:10.855652 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:57:10.855981 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:57:10.855985 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:57:10.856010 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:57:10.856368 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:10.856381 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:10.857637 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:10.857641 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:10.859378 (+ 1737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:10.859393 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:10.861010 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:10.861017 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:10.861173 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:10.861204 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 02:57:10.861294 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 02:57:10.861414 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":399,"threads_started":1}
I20260806 02:57:10.862284 21599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb6c47e130d645b4aab75a9d8297709b" candidate_uuid: "a60e56aca01147d2a3af980397d153c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c0c62cb384a4ec28aff24207d9deb3c" is_pre_election: true
I20260806 02:57:10.862800 21599 raft_consensus.cc:2454] T eb6c47e130d645b4aab75a9d8297709b P 4c0c62cb384a4ec28aff24207d9deb3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a60e56aca01147d2a3af980397d153c9 in term 0.
I20260806 02:57:10.862834 21399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb6c47e130d645b4aab75a9d8297709b" candidate_uuid: "a60e56aca01147d2a3af980397d153c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f949feba7e274b1ca2eda42b5f7ee930" is_pre_election: true
I20260806 02:57:10.863361 21399 raft_consensus.cc:2454] T eb6c47e130d645b4aab75a9d8297709b P f949feba7e274b1ca2eda42b5f7ee930 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a60e56aca01147d2a3af980397d153c9 in term 0.
I20260806 02:57:10.863456 21699 leader_election.cc:302] T eb6c47e130d645b4aab75a9d8297709b P a60e56aca01147d2a3af980397d153c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c0c62cb384a4ec28aff24207d9deb3c, a60e56aca01147d2a3af980397d153c9; no voters:
I20260806 02:57:10.863854 21905 raft_consensus.cc:2781] T eb6c47e130d645b4aab75a9d8297709b P a60e56aca01147d2a3af980397d153c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:57:10.863999 21905 raft_consensus.cc:482] T eb6c47e130d645b4aab75a9d8297709b P a60e56aca01147d2a3af980397d153c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:57:10.864104 21905 raft_consensus.cc:3037] T eb6c47e130d645b4aab75a9d8297709b P a60e56aca01147d2a3af980397d153c9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:57:10.866451 21905 raft_consensus.cc:504] T eb6c47e130d645b4aab75a9d8297709b P a60e56aca01147d2a3af980397d153c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a60e56aca01147d2a3af980397d153c9" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 37633 } } peers { permanent_uuid: "f949feba7e274b1ca2eda42b5f7ee930" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 45187 } } peers { permanent_uuid: "4c0c62cb384a4ec28aff24207d9deb3c" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 37419 } }
I20260806 02:57:10.867094 21905 leader_election.cc:288] T eb6c47e130d645b4aab75a9d8297709b P a60e56aca01147d2a3af980397d153c9 [CANDIDATE]: Term 1 election: Requested vote from peers f949feba7e274b1ca2eda42b5f7ee930 (127.25.211.129:45187), 4c0c62cb384a4ec28aff24207d9deb3c (127.25.211.130:37419)
I20260806 02:57:10.867672 21599 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb6c47e130d645b4aab75a9d8297709b" candidate_uuid: "a60e56aca01147d2a3af980397d153c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c0c62cb384a4ec28aff24207d9deb3c"
I20260806 02:57:10.867710 21399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb6c47e130d645b4aab75a9d8297709b" candidate_uuid: "a60e56aca01147d2a3af980397d153c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f949feba7e274b1ca2eda42b5f7ee930"
I20260806 02:57:10.867862 21599 raft_consensus.cc:3037] T eb6c47e130d645b4aab75a9d8297709b P 4c0c62cb384a4ec28aff24207d9deb3c [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:57:10.867894 21399 raft_consensus.cc:3037] T eb6c47e130d645b4aab75a9d8297709b P f949feba7e274b1ca2eda42b5f7ee930 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:57:10.870077 21399 raft_consensus.cc:2454] T eb6c47e130d645b4aab75a9d8297709b P f949feba7e274b1ca2eda42b5f7ee930 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a60e56aca01147d2a3af980397d153c9 in term 1.
I20260806 02:57:10.870078 21599 raft_consensus.cc:2454] T eb6c47e130d645b4aab75a9d8297709b P 4c0c62cb384a4ec28aff24207d9deb3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a60e56aca01147d2a3af980397d153c9 in term 1.
I20260806 02:57:10.870592 21699 leader_election.cc:302] T eb6c47e130d645b4aab75a9d8297709b P a60e56aca01147d2a3af980397d153c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c0c62cb384a4ec28aff24207d9deb3c, a60e56aca01147d2a3af980397d153c9; no voters:
I20260806 02:57:10.870913 21905 raft_consensus.cc:2781] T eb6c47e130d645b4aab75a9d8297709b P a60e56aca01147d2a3af980397d153c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:57:10.871399 21905 raft_consensus.cc:686] T eb6c47e130d645b4aab75a9d8297709b P a60e56aca01147d2a3af980397d153c9 [term 1 LEADER]: Becoming Leader. State: Replica: a60e56aca01147d2a3af980397d153c9, State: Running, Role: LEADER
I20260806 02:57:10.871871 21905 consensus_queue.cc:237] T eb6c47e130d645b4aab75a9d8297709b P a60e56aca01147d2a3af980397d153c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a60e56aca01147d2a3af980397d153c9" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 37633 } } peers { permanent_uuid: "f949feba7e274b1ca2eda42b5f7ee930" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 45187 } } peers { permanent_uuid: "4c0c62cb384a4ec28aff24207d9deb3c" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 37419 } }
I20260806 02:57:10.876828 21195 catalog_manager.cc:5495] T eb6c47e130d645b4aab75a9d8297709b P a60e56aca01147d2a3af980397d153c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a60e56aca01147d2a3af980397d153c9 (127.25.211.131). New cstate: current_term: 1 leader_uuid: "a60e56aca01147d2a3af980397d153c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a60e56aca01147d2a3af980397d153c9" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 37633 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f949feba7e274b1ca2eda42b5f7ee930" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 45187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c0c62cb384a4ec28aff24207d9deb3c" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 37419 } health_report { overall_health: UNKNOWN } } }
W20260806 02:57:10.880919 21864 tablet.cc:2369] T eb6c47e130d645b4aab75a9d8297709b P a60e56aca01147d2a3af980397d153c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:57:10.910476 21884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:10.904643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46228 (local address 127.25.211.131:37633)
0806 02:57:10.904843 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:10.904849 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:10.904985 (+ 136us) server_negotiation.cc:407] Connection header received
0806 02:57:10.905212 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:10.905216 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:10.905301 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:10.905445 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:57:10.905900 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:10.906689 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:10.908116 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:10.908544 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:10.908673 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:10.909286 (+ 613us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:57:10.909824 (+ 538us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:57:10.909989 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:10.910187 (+ 198us) server_negotiation.cc:299] Negotiation successful
0806 02:57:10.910240 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 02:57:10.919219 21399 raft_consensus.cc:1261] T eb6c47e130d645b4aab75a9d8297709b P f949feba7e274b1ca2eda42b5f7ee930 [term 1 FOLLOWER]: Refusing update from remote peer a60e56aca01147d2a3af980397d153c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:57:10.919219 21599 raft_consensus.cc:1261] T eb6c47e130d645b4aab75a9d8297709b P 4c0c62cb384a4ec28aff24207d9deb3c [term 1 FOLLOWER]: Refusing update from remote peer a60e56aca01147d2a3af980397d153c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:57:10.920399 21911 consensus_queue.cc:1035] T eb6c47e130d645b4aab75a9d8297709b P a60e56aca01147d2a3af980397d153c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f949feba7e274b1ca2eda42b5f7ee930" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 45187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:57:10.920816 21905 consensus_queue.cc:1035] T eb6c47e130d645b4aab75a9d8297709b P a60e56aca01147d2a3af980397d153c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c0c62cb384a4ec28aff24207d9deb3c" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 37419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:57:10.932300 21915 mvcc.cc:205] Tried to move back new op lower bound from 7315394686633689088 to 7315394686455259136. Current Snapshot: MvccSnapshot[applied={T|T < 7315394686633689088}]
I20260806 02:57:10.935179 21920 mvcc.cc:205] Tried to move back new op lower bound from 7315394686633689088 to 7315394686455259136. Current Snapshot: MvccSnapshot[applied={T|T < 7315394686633689088}]
I20260806 02:57:10.949541 21197 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:57:10.949831 21197 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:57:10.953676 21197 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 950e0370b249408cb9ab5111ea96ce92: Sending DeleteTablet for 3 replicas of tablet eb6c47e130d645b4aab75a9d8297709b
I20260806 02:57:10.955114 21765 tablet_service.cc:1514] Processing DeleteTablet for tablet eb6c47e130d645b4aab75a9d8297709b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:57:10 UTC) from {username='kudu', principal='kudu/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:46226
I20260806 02:57:10.955274 21571 tablet_service.cc:1514] Processing DeleteTablet for tablet eb6c47e130d645b4aab75a9d8297709b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:57:10 UTC) from {username='kudu', principal='kudu/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:46834
I20260806 02:57:10.956677 21372 tablet_service.cc:1514] Processing DeleteTablet for tablet eb6c47e130d645b4aab75a9d8297709b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:57:10 UTC) from {username='kudu', principal='kudu/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:39110
I20260806 02:57:10.957496 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 21283
I20260806 02:57:10.965159 21933 tablet_replica.cc:330] T eb6c47e130d645b4aab75a9d8297709b P f949feba7e274b1ca2eda42b5f7ee930: stopping tablet replica
I20260806 02:57:10.965770 21933 raft_consensus.cc:2229] T eb6c47e130d645b4aab75a9d8297709b P f949feba7e274b1ca2eda42b5f7ee930 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:57:10.966423 21933 raft_consensus.cc:2258] T eb6c47e130d645b4aab75a9d8297709b P f949feba7e274b1ca2eda42b5f7ee930 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:57:10.970901 21933 ts_tablet_manager.cc:1905] T eb6c47e130d645b4aab75a9d8297709b P f949feba7e274b1ca2eda42b5f7ee930: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:57:10.973748 21932 tablet_replica.cc:330] T eb6c47e130d645b4aab75a9d8297709b P a60e56aca01147d2a3af980397d153c9: stopping tablet replica
I20260806 02:57:10.974581 21930 tablet_replica.cc:330] T eb6c47e130d645b4aab75a9d8297709b P 4c0c62cb384a4ec28aff24207d9deb3c: stopping tablet replica
I20260806 02:57:10.975072 21930 raft_consensus.cc:2229] T eb6c47e130d645b4aab75a9d8297709b P 4c0c62cb384a4ec28aff24207d9deb3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:57:10.975713 21930 raft_consensus.cc:2258] T eb6c47e130d645b4aab75a9d8297709b P 4c0c62cb384a4ec28aff24207d9deb3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:57:10.976365 21932 raft_consensus.cc:2229] T eb6c47e130d645b4aab75a9d8297709b P a60e56aca01147d2a3af980397d153c9 [term 1 LEADER]: Raft consensus shutting down.
Aug 06 02:57:10 dist-test-slave-zj8q krb5kdc[21130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.129@KRBTEST.COM
I20260806 02:57:10.977233 21932 raft_consensus.cc:2258] T eb6c47e130d645b4aab75a9d8297709b P a60e56aca01147d2a3af980397d153c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:57:10.980162 21930 ts_tablet_manager.cc:1905] T eb6c47e130d645b4aab75a9d8297709b P 4c0c62cb384a4ec28aff24207d9deb3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:57:10.981460 21933 ts_tablet_manager.cc:1918] T eb6c47e130d645b4aab75a9d8297709b P f949feba7e274b1ca2eda42b5f7ee930: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:57:10.981674 21933 log.cc:1196] T eb6c47e130d645b4aab75a9d8297709b P f949feba7e274b1ca2eda42b5f7ee930: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/ts-0/wal/wals/eb6c47e130d645b4aab75a9d8297709b
I20260806 02:57:10.982225 21933 ts_tablet_manager.cc:1939] T eb6c47e130d645b4aab75a9d8297709b P f949feba7e274b1ca2eda42b5f7ee930: Deleting consensus metadata
I20260806 02:57:10.984526 21172 catalog_manager.cc:4841] TS f949feba7e274b1ca2eda42b5f7ee930 (127.25.211.129:45187): tablet eb6c47e130d645b4aab75a9d8297709b (table test-table [id=b70afa7612f0498c8aaa642272986ca1]) successfully deleted
I20260806 02:57:10.985961 21932 ts_tablet_manager.cc:1905] T eb6c47e130d645b4aab75a9d8297709b P a60e56aca01147d2a3af980397d153c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:57:10.987475 21888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:10.959081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39124 (local address 127.25.211.129:45187)
0806 02:57:10.959277 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:10.959283 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:10.964168 (+ 4885us) server_negotiation.cc:407] Connection header received
0806 02:57:10.964373 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:10.964378 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:10.964476 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:10.964605 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:10.967534 (+ 2929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:10.968695 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:10.970735 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:10.971462 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:10.971582 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:10.978437 (+ 6855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:10.978458 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:10.978481 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:57:10.978549 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:10.983004 (+ 4455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:10.984403 (+ 1399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:10.984414 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:10.984417 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:10.984604 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:10.985234 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:10.985240 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:10.985243 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:10.985786 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:10.986196 (+ 410us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:10.986831 (+ 635us) server_negotiation.cc:299] Negotiation successful
0806 02:57:10.987080 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 02:57:10.988615 21930 ts_tablet_manager.cc:1918] T eb6c47e130d645b4aab75a9d8297709b P 4c0c62cb384a4ec28aff24207d9deb3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:57:10.988811 21930 log.cc:1196] T eb6c47e130d645b4aab75a9d8297709b P 4c0c62cb384a4ec28aff24207d9deb3c: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/ts-1/wal/wals/eb6c47e130d645b4aab75a9d8297709b
I20260806 02:57:10.989326 21457 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:57:10.989336 21930 ts_tablet_manager.cc:1939] T eb6c47e130d645b4aab75a9d8297709b P 4c0c62cb384a4ec28aff24207d9deb3c: Deleting consensus metadata
I20260806 02:57:10.991452 21173 catalog_manager.cc:4841] TS 4c0c62cb384a4ec28aff24207d9deb3c (127.25.211.130:37419): tablet eb6c47e130d645b4aab75a9d8297709b (table test-table [id=b70afa7612f0498c8aaa642272986ca1]) successfully deleted
I20260806 02:57:10.995237 21932 ts_tablet_manager.cc:1918] T eb6c47e130d645b4aab75a9d8297709b P a60e56aca01147d2a3af980397d153c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:57:10.995425 21932 log.cc:1196] T eb6c47e130d645b4aab75a9d8297709b P a60e56aca01147d2a3af980397d153c9: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1785984707115702-26446-0/minicluster-data/ts-2/wal/wals/eb6c47e130d645b4aab75a9d8297709b
I20260806 02:57:10.995918 21932 ts_tablet_manager.cc:1939] T eb6c47e130d645b4aab75a9d8297709b P a60e56aca01147d2a3af980397d153c9: Deleting consensus metadata
I20260806 02:57:10.997535 21172 catalog_manager.cc:4841] TS a60e56aca01147d2a3af980397d153c9 (127.25.211.131:37633): tablet eb6c47e130d645b4aab75a9d8297709b (table test-table [id=b70afa7612f0498c8aaa642272986ca1]) successfully deleted
I20260806 02:57:11.114197 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 21283
I20260806 02:57:11.137326 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 21474
Aug 06 02:57:11 dist-test-slave-zj8q krb5kdc[21130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.130@KRBTEST.COM
I20260806 02:57:11.154755 21887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:11.138213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46838 (local address 127.25.211.130:37419)
0806 02:57:11.138386 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:11.138392 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:11.138414 (+ 22us) server_negotiation.cc:407] Connection header received
0806 02:57:11.138669 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:11.138676 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:11.138764 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:11.138864 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:11.140638 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:11.141400 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:11.143330 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:11.143872 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:11.144000 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:11.148726 (+ 4726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:11.148745 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:11.148764 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:57:11.148809 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:11.151757 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:11.152517 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:11.152525 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:11.152528 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:11.152676 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:11.153087 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:11.153090 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:11.153093 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:11.153509 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:11.153803 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:11.154193 (+ 390us) server_negotiation.cc:299] Negotiation successful
0806 02:57:11.154419 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 02:57:11.155536 21656 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:57:11.245079 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 21474
I20260806 02:57:11.276844 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 21674
Aug 06 02:57:11 dist-test-slave-zj8q krb5kdc[21130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.131@KRBTEST.COM
I20260806 02:57:11.293665 21884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:11.277612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46232 (local address 127.25.211.131:37633)
0806 02:57:11.277823 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:11.277828 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:11.277850 (+ 22us) server_negotiation.cc:407] Connection header received
0806 02:57:11.277983 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:11.277988 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:11.278054 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:11.278140 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:11.279607 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:11.280421 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:11.281762 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:11.282265 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:11.282387 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:11.287350 (+ 4963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:11.287366 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:11.287384 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:57:11.287432 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:11.290595 (+ 3163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:11.291385 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:11.291393 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:11.291395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:11.291569 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:11.292007 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:11.292010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:11.292013 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:11.292396 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:11.292722 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:11.293034 (+ 312us) server_negotiation.cc:299] Negotiation successful
0806 02:57:11.293261 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260806 02:57:11.294520 21855 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:57:11.417974 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 21674
I20260806 02:57:11.444785 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 21146
I20260806 02:57:11.445978 21238 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:57:11.558861 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 21146
2026-08-06T02:57:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4707 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-06T02:57:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:57:11Z Disabled control of system clock
I20260806 02:57:11.661718 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:39991
--webserver_interface=127.25.211.190
--webserver_port=0
--builtin_ntp_servers=127.25.211.148:42539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.211.190:39991
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 02:57:12.017647 21944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:57:12.018105 21944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:57:12.018277 21944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:57:12.027108 21944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:57:12.027218 21944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:57:12.027264 21944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:57:12.027320 21944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:57:12.027371 21944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:57:12.037009 21944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:42539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.211.190:39991
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:39991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.211.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:57:12.038424 21944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:57:12.040333 21944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:57:12.052055 21956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:12.052129 21955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:12.052073 21959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:57:12.053107 21944 server_base.cc:1034] running on GCE node
I20260806 02:57:12.053942 21944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:57:12.055526 21944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:57:12.056771 21944 hybrid_clock.cc:648] HybridClock initialized: now 1785985032056695 us; error 54 us; skew 500 ppm
I20260806 02:57:12.057147 21944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
W20260806 02:57:13.018882 21944 net_util.cc:504] Time spent look up canonical hostname for localhost 'dist-test-slave-zj8q': real 0.961s user 0.001s sys 0.000s
I20260806 02:57:13.023612 21944 webserver.cc:460] Webserver started at http://127.25.211.190:37647/ using document root <none> and password file <none>
I20260806 02:57:13.024406 21944 fs_manager.cc:359] Metadata directory not provided
I20260806 02:57:13.024544 21944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:57:13.024873 21944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:57:13.028003 21944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/master-0/data/instance:
uuid: "09bf27ac30694e14884d890e2381a329"
format_stamp: "Formatted at 2026-08-06 02:57:13 on dist-test-slave-zj8q"
server_key: "c9c10c7f365bf6277ce1998a21851ec8"
server_key_iv: "bc32421fa77b24362104831caad5828b"
server_key_version: "encryptionkey@0"
I20260806 02:57:13.028824 21944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/master-0/wal/instance:
uuid: "09bf27ac30694e14884d890e2381a329"
format_stamp: "Formatted at 2026-08-06 02:57:13 on dist-test-slave-zj8q"
server_key: "c9c10c7f365bf6277ce1998a21851ec8"
server_key_iv: "bc32421fa77b24362104831caad5828b"
server_key_version: "encryptionkey@0"
I20260806 02:57:13.034220 21944 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260806 02:57:13.037783 21966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:13.039186 21944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260806 02:57:13.039376 21944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/master-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/master-0/wal
uuid: "09bf27ac30694e14884d890e2381a329"
format_stamp: "Formatted at 2026-08-06 02:57:13 on dist-test-slave-zj8q"
server_key: "c9c10c7f365bf6277ce1998a21851ec8"
server_key_iv: "bc32421fa77b24362104831caad5828b"
server_key_version: "encryptionkey@0"
I20260806 02:57:13.039579 21944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:57:13.062476 21944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:57:13.063395 21944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:57:13.063715 21944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:57:13.082762 21944 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.190:39991
I20260806 02:57:13.082796 22042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.190:39991 every 8 connection(s)
I20260806 02:57:13.084174 21944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
I20260806 02:57:13.085383 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 21944
I20260806 02:57:13.085584 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/master-0/wal/instance
I20260806 02:57:13.086201 26446 external_mini_cluster.cc:1428] Setting key e3eb26551c71dc0d56cbb3a00baf34e2
I20260806 02:57:13.089018 22043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:57:13.100718 22047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:13.088251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53106 (local address 127.25.211.190:39991)
0806 02:57:13.089405 (+ 1154us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:13.089418 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:13.089488 (+ 70us) server_negotiation.cc:407] Connection header received
0806 02:57:13.091457 (+ 1969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:13.091479 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:13.091942 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:13.092303 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:13.092873 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:13.094330 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:13.096248 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:13.096947 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:13.097106 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:13.098373 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:13.098380 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:13.098397 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:57:13.098974 (+ 577us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:13.099070 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:57:13.099093 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:13.099189 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:13.099415 (+ 226us) server_negotiation.cc:299] Negotiation successful
0806 02:57:13.099749 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":356,"threads_started":1}
I20260806 02:57:13.101672 22043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09bf27ac30694e14884d890e2381a329: Bootstrap starting.
I20260806 02:57:13.104831 22043 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09bf27ac30694e14884d890e2381a329: Neither blocks nor log segments found. Creating new log.
I20260806 02:57:13.105846 22043 log.cc:824] T 00000000000000000000000000000000 P 09bf27ac30694e14884d890e2381a329: Log is configured to *not* fsync() on all Append() calls
I20260806 02:57:13.108842 22043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09bf27ac30694e14884d890e2381a329: No bootstrap required, opened a new log
I20260806 02:57:13.116849 22043 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09bf27ac30694e14884d890e2381a329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09bf27ac30694e14884d890e2381a329" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 39991 } }
I20260806 02:57:13.117286 22043 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09bf27ac30694e14884d890e2381a329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:57:13.117385 22043 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09bf27ac30694e14884d890e2381a329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09bf27ac30694e14884d890e2381a329, State: Initialized, Role: FOLLOWER
I20260806 02:57:13.117996 22043 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09bf27ac30694e14884d890e2381a329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09bf27ac30694e14884d890e2381a329" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 39991 } }
I20260806 02:57:13.118296 22043 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09bf27ac30694e14884d890e2381a329 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:57:13.118435 22043 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09bf27ac30694e14884d890e2381a329 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:57:13.118605 22043 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09bf27ac30694e14884d890e2381a329 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:57:13.121292 22043 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09bf27ac30694e14884d890e2381a329 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09bf27ac30694e14884d890e2381a329" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 39991 } }
I20260806 02:57:13.121842 22043 leader_election.cc:302] T 00000000000000000000000000000000 P 09bf27ac30694e14884d890e2381a329 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09bf27ac30694e14884d890e2381a329; no voters:
I20260806 02:57:13.122466 22043 leader_election.cc:288] T 00000000000000000000000000000000 P 09bf27ac30694e14884d890e2381a329 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:57:13.122598 22051 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09bf27ac30694e14884d890e2381a329 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:57:13.123309 22051 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09bf27ac30694e14884d890e2381a329 [term 1 LEADER]: Becoming Leader. State: Replica: 09bf27ac30694e14884d890e2381a329, State: Running, Role: LEADER
I20260806 02:57:13.123919 22051 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09bf27ac30694e14884d890e2381a329 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09bf27ac30694e14884d890e2381a329" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 39991 } }
I20260806 02:57:13.124550 22043 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09bf27ac30694e14884d890e2381a329 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:57:13.126782 22054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09bf27ac30694e14884d890e2381a329 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09bf27ac30694e14884d890e2381a329" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09bf27ac30694e14884d890e2381a329" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 39991 } } }
I20260806 02:57:13.127192 22054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09bf27ac30694e14884d890e2381a329 [sys.catalog]: This master's current role is: LEADER
I20260806 02:57:13.127619 22055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09bf27ac30694e14884d890e2381a329 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09bf27ac30694e14884d890e2381a329. Latest consensus state: current_term: 1 leader_uuid: "09bf27ac30694e14884d890e2381a329" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09bf27ac30694e14884d890e2381a329" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 39991 } } }
I20260806 02:57:13.127918 22055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09bf27ac30694e14884d890e2381a329 [sys.catalog]: This master's current role is: LEADER
I20260806 02:57:13.131714 22060 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:57:13.137519 22060 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:57:13.146968 22060 catalog_manager.cc:1347] Generated new cluster ID: dedf7f7c2ac44dc39b29fc8f44fbd233
I20260806 02:57:13.147115 22060 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:57:13.161123 22060 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:57:13.162515 22060 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:57:13.172518 22060 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09bf27ac30694e14884d890e2381a329: Generated new TSK 0
I20260806 02:57:13.173355 22060 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:57:13.189679 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.129:0
--local_ip_for_outbound_sockets=127.25.211.129
--webserver_interface=127.25.211.129
--webserver_port=0
--tserver_master_addrs=127.25.211.190:39991
--builtin_ntp_servers=127.25.211.148:42539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 02:57:13.541556 22075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:57:13.541841 22075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:57:13.541954 22075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:57:13.549333 22075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:57:13.549484 22075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:57:13.549631 22075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.129
I20260806 02:57:13.558887 22075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:42539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.211.129
--webserver_port=0
--tserver_master_addrs=127.25.211.190:39991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.129
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:57:13.560346 22075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:57:13.562493 22075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:57:13.576678 22083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:13.576766 22081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:13.577049 22086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:57:13.577997 22075 server_base.cc:1034] running on GCE node
I20260806 02:57:13.578765 22075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:57:13.579867 22075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:57:13.581164 22075 hybrid_clock.cc:648] HybridClock initialized: now 1785985033581102 us; error 53 us; skew 500 ppm
I20260806 02:57:13.581538 22075 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:57:13.584131 22075 webserver.cc:460] Webserver started at http://127.25.211.129:35107/ using document root <none> and password file <none>
I20260806 02:57:13.584859 22075 fs_manager.cc:359] Metadata directory not provided
I20260806 02:57:13.584970 22075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:57:13.585292 22075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:57:13.588394 22075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/ts-0/data/instance:
uuid: "5b1b933eb9534a6391f00183d1f07827"
format_stamp: "Formatted at 2026-08-06 02:57:13 on dist-test-slave-zj8q"
server_key: "0bdfe364519662a60d6bc32bb1d661d5"
server_key_iv: "0a7583a9de7da3ef7cef2b89df297e98"
server_key_version: "encryptionkey@0"
I20260806 02:57:13.589259 22075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/ts-0/wal/instance:
uuid: "5b1b933eb9534a6391f00183d1f07827"
format_stamp: "Formatted at 2026-08-06 02:57:13 on dist-test-slave-zj8q"
server_key: "0bdfe364519662a60d6bc32bb1d661d5"
server_key_iv: "0a7583a9de7da3ef7cef2b89df297e98"
server_key_version: "encryptionkey@0"
I20260806 02:57:13.594830 22075 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260806 02:57:13.598482 22092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:13.599911 22075 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:57:13.600088 22075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/ts-0/wal
uuid: "5b1b933eb9534a6391f00183d1f07827"
format_stamp: "Formatted at 2026-08-06 02:57:13 on dist-test-slave-zj8q"
server_key: "0bdfe364519662a60d6bc32bb1d661d5"
server_key_iv: "0a7583a9de7da3ef7cef2b89df297e98"
server_key_version: "encryptionkey@0"
I20260806 02:57:13.600302 22075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:57:13.612880 22075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:57:13.613704 22075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:57:13.613994 22075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:57:13.615022 22075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:57:13.616479 22075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:57:13.616582 22075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:13.616703 22075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:57:13.616772 22075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:13.647729 22075 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.129:39831
I20260806 02:57:13.647805 22259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.129:39831 every 8 connection(s)
I20260806 02:57:13.649130 22075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
I20260806 02:57:13.653929 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 22075
I20260806 02:57:13.654129 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/ts-0/wal/instance
I20260806 02:57:13.654702 26446 external_mini_cluster.cc:1428] Setting key 21f5c94e7bbc488c2741e9019bfc4bff
I20260806 02:57:13.657958 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.130:0
--local_ip_for_outbound_sockets=127.25.211.130
--webserver_interface=127.25.211.130
--webserver_port=0
--tserver_master_addrs=127.25.211.190:39991
--builtin_ntp_servers=127.25.211.148:42539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 02:57:13.663743 22263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:13.652063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:58163 (local address 127.25.211.190:39991)
0806 02:57:13.652537 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:13.652542 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:13.653773 (+ 1231us) server_negotiation.cc:407] Connection header received
0806 02:57:13.656107 (+ 2334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:13.656111 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:13.656182 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:13.656280 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:13.657755 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:13.658590 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:13.660809 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:13.661265 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:13.661378 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:13.662991 (+ 1613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:13.662996 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:13.663000 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:57:13.663035 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:13.663079 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:57:13.663092 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:13.663167 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:13.663398 (+ 231us) server_negotiation.cc:299] Negotiation successful
0806 02:57:13.663484 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":241,"threads_started":1}
I20260806 02:57:13.665349 22264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:13.652574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:39991 (local address 127.25.211.129:58163)
0806 02:57:13.653562 (+ 988us) negotiation.cc:107] Waiting for socket to connect
0806 02:57:13.653622 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0806 02:57:13.655788 (+ 2166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:57:13.656564 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:57:13.656575 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:57:13.656667 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:57:13.657544 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:13.657574 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:13.658758 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:13.658764 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:13.660634 (+ 1870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:13.660650 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:13.661516 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:13.661523 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:13.661810 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:13.662604 (+ 794us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:57:13.662607 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:57:13.662664 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:57:13.662772 (+ 108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:57:13.663189 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:57:13.663200 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:57:13.663208 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 02:57:13.663395 (+ 187us) client_negotiation.cc:240] Negotiation successful
0806 02:57:13.663812 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":294,"threads_started":1}
I20260806 02:57:13.667222 22260 heartbeater.cc:347] Connected to a master server at 127.25.211.190:39991
I20260806 02:57:13.667578 22260 heartbeater.cc:464] Registering TS with master...
I20260806 02:57:13.668555 22260 heartbeater.cc:511] Master 127.25.211.190:39991 requested a full tablet report, sending...
I20260806 02:57:13.670987 21991 ts_manager.cc:194] Registered new tserver with Master: 5b1b933eb9534a6391f00183d1f07827 (127.25.211.129:39831)
I20260806 02:57:13.672941 21991 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.211.129:58163
W20260806 02:57:14.001650 22265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:57:14.001962 22265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:57:14.002050 22265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:57:14.009960 22265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:57:14.010092 22265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:57:14.010228 22265 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.130
I20260806 02:57:14.021639 22265 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:42539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.211.130
--webserver_port=0
--tserver_master_addrs=127.25.211.190:39991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.130
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:57:14.023020 22265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:57:14.024823 22265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:57:14.038198 22272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:14.038189 22273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:14.038112 22277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:57:14.038916 22265 server_base.cc:1034] running on GCE node
I20260806 02:57:14.039535 22265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:57:14.040455 22265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:57:14.041662 22265 hybrid_clock.cc:648] HybridClock initialized: now 1785985034041618 us; error 41 us; skew 500 ppm
I20260806 02:57:14.041980 22265 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:57:14.044312 22265 webserver.cc:460] Webserver started at http://127.25.211.130:34053/ using document root <none> and password file <none>
I20260806 02:57:14.045008 22265 fs_manager.cc:359] Metadata directory not provided
I20260806 02:57:14.045099 22265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:57:14.045413 22265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:57:14.048327 22265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/ts-1/data/instance:
uuid: "1474992d186d4677b0ff0989920739b5"
format_stamp: "Formatted at 2026-08-06 02:57:14 on dist-test-slave-zj8q"
server_key: "33532194d8db67d26fc88a876a5972a8"
server_key_iv: "1a5af7a70de99a25a35bb52d4426ea65"
server_key_version: "encryptionkey@0"
I20260806 02:57:14.049214 22265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/ts-1/wal/instance:
uuid: "1474992d186d4677b0ff0989920739b5"
format_stamp: "Formatted at 2026-08-06 02:57:14 on dist-test-slave-zj8q"
server_key: "33532194d8db67d26fc88a876a5972a8"
server_key_iv: "1a5af7a70de99a25a35bb52d4426ea65"
server_key_version: "encryptionkey@0"
I20260806 02:57:14.054533 22265 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260806 02:57:14.058234 22285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:14.059646 22265 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:57:14.059813 22265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/ts-1/wal
uuid: "1474992d186d4677b0ff0989920739b5"
format_stamp: "Formatted at 2026-08-06 02:57:14 on dist-test-slave-zj8q"
server_key: "33532194d8db67d26fc88a876a5972a8"
server_key_iv: "1a5af7a70de99a25a35bb52d4426ea65"
server_key_version: "encryptionkey@0"
I20260806 02:57:14.060019 22265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:57:14.084832 22265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:57:14.085709 22265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:57:14.086010 22265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:57:14.087147 22265 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:57:14.088651 22265 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:57:14.088752 22265 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:14.088857 22265 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:57:14.088929 22265 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:14.119982 22265 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.130:40011
I20260806 02:57:14.120051 22447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.130:40011 every 8 connection(s)
I20260806 02:57:14.121304 22265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
I20260806 02:57:14.123016 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 22265
I20260806 02:57:14.123207 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/ts-1/wal/instance
I20260806 02:57:14.123683 26446 external_mini_cluster.cc:1428] Setting key 19790bbef2f14df845e2a0ad40735882
I20260806 02:57:14.127015 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.131:0
--local_ip_for_outbound_sockets=127.25.211.131
--webserver_interface=127.25.211.131
--webserver_port=0
--tserver_master_addrs=127.25.211.190:39991
--builtin_ntp_servers=127.25.211.148:42539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 02:57:14.135996 22263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:14.124024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:37881 (local address 127.25.211.190:39991)
0806 02:57:14.124257 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:14.124263 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:14.125907 (+ 1644us) server_negotiation.cc:407] Connection header received
0806 02:57:14.128428 (+ 2521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:14.128433 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:14.128507 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:14.128592 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:14.130069 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:14.130865 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:14.133061 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:14.133482 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:14.133603 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:14.135235 (+ 1632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:14.135240 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:14.135243 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:57:14.135263 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:14.135299 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:57:14.135312 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:14.135383 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:14.135625 (+ 242us) server_negotiation.cc:299] Negotiation successful
0806 02:57:14.135715 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 02:57:14.137375 22454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:14.124644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:39991 (local address 127.25.211.130:37881)
0806 02:57:14.125748 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0806 02:57:14.125789 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 02:57:14.128096 (+ 2307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:57:14.128875 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:57:14.128888 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:57:14.129022 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:57:14.129854 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:14.129879 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:14.131032 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:14.131037 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:14.132869 (+ 1832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:14.132885 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:14.133738 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:14.133745 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:14.134062 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:14.134848 (+ 786us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:57:14.134852 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:57:14.134908 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:57:14.135024 (+ 116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:57:14.135403 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:57:14.135414 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:57:14.135423 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 02:57:14.135611 (+ 188us) client_negotiation.cc:240] Negotiation successful
0806 02:57:14.136065 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":326,"threads_started":1}
I20260806 02:57:14.138952 22450 heartbeater.cc:347] Connected to a master server at 127.25.211.190:39991
I20260806 02:57:14.139307 22450 heartbeater.cc:464] Registering TS with master...
I20260806 02:57:14.140234 22450 heartbeater.cc:511] Master 127.25.211.190:39991 requested a full tablet report, sending...
I20260806 02:57:14.141929 21991 ts_manager.cc:194] Registered new tserver with Master: 1474992d186d4677b0ff0989920739b5 (127.25.211.130:40011)
I20260806 02:57:14.142918 21991 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.211.130:37881
W20260806 02:57:14.475348 22455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:57:14.475659 22455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:57:14.475745 22455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:57:14.482987 22455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:57:14.483104 22455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:57:14.483223 22455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.131
I20260806 02:57:14.492388 22455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:42539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.211.131
--webserver_port=0
--tserver_master_addrs=127.25.211.190:39991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.131
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:57:14.493710 22455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:57:14.495483 22455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:57:14.508599 22466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:14.508512 22469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:14.508531 22465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:57:14.509398 22455 server_base.cc:1034] running on GCE node
I20260806 02:57:14.510035 22455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:57:14.511041 22455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:57:14.512269 22455 hybrid_clock.cc:648] HybridClock initialized: now 1785985034512210 us; error 50 us; skew 500 ppm
I20260806 02:57:14.512620 22455 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:57:14.514923 22455 webserver.cc:460] Webserver started at http://127.25.211.131:33811/ using document root <none> and password file <none>
I20260806 02:57:14.515594 22455 fs_manager.cc:359] Metadata directory not provided
I20260806 02:57:14.515697 22455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:57:14.516009 22455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:57:14.518977 22455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/ts-2/data/instance:
uuid: "110d7b90e2ce4c98a94d339c8a522b89"
format_stamp: "Formatted at 2026-08-06 02:57:14 on dist-test-slave-zj8q"
server_key: "2a9a1af4d53041fcb014598d3d8782d1"
server_key_iv: "88384c2c42ff4ec4b2833abf5a8b61ff"
server_key_version: "encryptionkey@0"
I20260806 02:57:14.519765 22455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/ts-2/wal/instance:
uuid: "110d7b90e2ce4c98a94d339c8a522b89"
format_stamp: "Formatted at 2026-08-06 02:57:14 on dist-test-slave-zj8q"
server_key: "2a9a1af4d53041fcb014598d3d8782d1"
server_key_iv: "88384c2c42ff4ec4b2833abf5a8b61ff"
server_key_version: "encryptionkey@0"
I20260806 02:57:14.525287 22455 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260806 02:57:14.529422 22477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:14.530885 22455 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:57:14.531056 22455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/ts-2/wal
uuid: "110d7b90e2ce4c98a94d339c8a522b89"
format_stamp: "Formatted at 2026-08-06 02:57:14 on dist-test-slave-zj8q"
server_key: "2a9a1af4d53041fcb014598d3d8782d1"
server_key_iv: "88384c2c42ff4ec4b2833abf5a8b61ff"
server_key_version: "encryptionkey@0"
I20260806 02:57:14.531271 22455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:57:14.559717 22455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:57:14.560639 22455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:57:14.560941 22455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:57:14.562054 22455 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:57:14.563647 22455 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:57:14.563738 22455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:14.563850 22455 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:57:14.563917 22455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:14.595089 22455 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.131:39405
I20260806 02:57:14.595109 22657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.131:39405 every 8 connection(s)
I20260806 02:57:14.596398 22455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
I20260806 02:57:14.605500 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 22455
I20260806 02:57:14.605682 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/ts-2/wal/instance
I20260806 02:57:14.606139 26446 external_mini_cluster.cc:1428] Setting key 00b030deff1a6bd69a3e73a717ada8fb
I20260806 02:57:14.609671 22263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:14.599073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:39733 (local address 127.25.211.190:39991)
0806 02:57:14.599282 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:14.599287 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:14.600710 (+ 1423us) server_negotiation.cc:407] Connection header received
0806 02:57:14.602646 (+ 1936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:14.602653 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:14.602737 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:14.602831 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:14.604006 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:14.604807 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:14.606642 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:14.607245 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:14.607372 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:14.608877 (+ 1505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:14.608884 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:14.608888 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:57:14.608911 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:14.608955 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:57:14.608973 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:14.609058 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:14.609324 (+ 266us) server_negotiation.cc:299] Negotiation successful
0806 02:57:14.609418 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260806 02:57:14.610846 22661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:14.599593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:39991 (local address 127.25.211.131:39733)
0806 02:57:14.600540 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0806 02:57:14.600577 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 02:57:14.602325 (+ 1748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:57:14.603121 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:57:14.603133 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:57:14.603233 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:57:14.603824 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:14.603842 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:14.604974 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:14.604980 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:14.606490 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:14.606504 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:14.607500 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:14.607509 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:14.607749 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:14.608485 (+ 736us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:57:14.608488 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:57:14.608565 (+ 77us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:57:14.608677 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:57:14.609078 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:57:14.609091 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:57:14.609099 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 02:57:14.609299 (+ 200us) client_negotiation.cc:240] Negotiation successful
0806 02:57:14.609639 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":286,"threads_started":1}
I20260806 02:57:14.612233 22658 heartbeater.cc:347] Connected to a master server at 127.25.211.190:39991
I20260806 02:57:14.612588 22658 heartbeater.cc:464] Registering TS with master...
I20260806 02:57:14.613396 22658 heartbeater.cc:511] Master 127.25.211.190:39991 requested a full tablet report, sending...
I20260806 02:57:14.615075 21991 ts_manager.cc:194] Registered new tserver with Master: 110d7b90e2ce4c98a94d339c8a522b89 (127.25.211.131:39405)
I20260806 02:57:14.615866 21991 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.211.131:39733
I20260806 02:57:14.621744 26446 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:57:14.632663 22263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:14.626279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53110 (local address 127.25.211.190:39991)
0806 02:57:14.626481 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:14.626488 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:14.626910 (+ 422us) server_negotiation.cc:407] Connection header received
0806 02:57:14.627083 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:14.627087 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:14.627182 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:14.627268 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:14.627786 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:14.628526 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:14.629998 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:14.630549 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:14.630661 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:14.631913 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:14.631918 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:14.631921 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:57:14.631939 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:14.631973 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:57:14.631985 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:14.632120 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:14.632300 (+ 180us) server_negotiation.cc:299] Negotiation successful
0806 02:57:14.632358 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 02:57:14.637001 21991 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53110:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:57:14.644049 21991 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:57:14.676378 22260 heartbeater.cc:503] Master 127.25.211.190:39991 was elected leader, sending a full tablet report...
I20260806 02:57:14.682931 22675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:14.656857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.131:39405 (local address 127.0.0.1:42896)
0806 02:57:14.657691 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0806 02:57:14.657741 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 02:57:14.657915 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:57:14.675778 (+ 17863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:57:14.675783 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:57:14.675808 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:57:14.676136 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:14.676149 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:14.677944 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:14.677949 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:14.680012 (+ 2063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:14.680027 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:14.681871 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:14.681878 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:14.682044 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:14.682119 (+ 75us) client_negotiation.cc:769] Sending connection context
0806 02:57:14.682256 (+ 137us) client_negotiation.cc:240] Negotiation successful
0806 02:57:14.682508 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"mutex_wait_us":42,"thread_start_us":443,"threads_started":1}
I20260806 02:57:14.683133 22682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:14.657292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:39831 (local address 127.0.0.1:50860)
0806 02:57:14.671842 (+ 14550us) negotiation.cc:107] Waiting for socket to connect
0806 02:57:14.671877 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0806 02:57:14.672079 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:57:14.674395 (+ 2316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:57:14.674404 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:57:14.674447 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:57:14.675144 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:14.675160 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:14.677299 (+ 2139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:14.677304 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:14.680124 (+ 2820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:14.680140 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:14.682288 (+ 2148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:14.682294 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:14.682470 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:14.682501 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 02:57:14.682573 (+ 72us) client_negotiation.cc:240] Negotiation successful
0806 02:57:14.682713 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14326,"mutex_wait_us":172,"spinlock_wait_cycles":896,"thread_start_us":15928,"threads_started":1}
I20260806 02:57:14.683221 22677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:14.656985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42896 (local address 127.25.211.131:39405)
0806 02:57:14.675305 (+ 18320us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:14.675313 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:14.675346 (+ 33us) server_negotiation.cc:407] Connection header received
0806 02:57:14.675423 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:14.675429 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:14.675599 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:14.675762 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:57:14.676335 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:14.677785 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:14.680223 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:14.681637 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:14.682470 (+ 833us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:14.682663 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:14.682731 (+ 68us) server_negotiation.cc:299] Negotiation successful
0806 02:57:14.682903 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18145,"thread_start_us":17977,"threads_started":1}
I20260806 02:57:14.683844 22680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:14.657821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50860 (local address 127.25.211.129:39831)
0806 02:57:14.673271 (+ 15450us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:14.673278 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:14.673315 (+ 37us) server_negotiation.cc:407] Connection header received
0806 02:57:14.673435 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:14.673444 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:14.673772 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:14.674069 (+ 297us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:57:14.675360 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:14.677111 (+ 1751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:14.680374 (+ 3263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:14.681765 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:14.681938 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:14.682136 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:14.683469 (+ 1333us) server_negotiation.cc:299] Negotiation successful
0806 02:57:14.683605 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15267,"thread_start_us":15155,"threads_started":1}
I20260806 02:57:14.684019 22683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:14.658472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37002 (local address 127.25.211.130:40011)
0806 02:57:14.672821 (+ 14349us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:14.672829 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:14.673533 (+ 704us) server_negotiation.cc:407] Connection header received
0806 02:57:14.673625 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:14.673631 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:14.673810 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:14.674004 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:57:14.675401 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:14.677237 (+ 1836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:14.680494 (+ 3257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:14.681472 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:14.681626 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:14.681870 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:14.683487 (+ 1617us) server_negotiation.cc:299] Negotiation successful
0806 02:57:14.683670 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14194,"thread_start_us":14087,"threads_started":1}
I20260806 02:57:14.684700 22678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:14.658096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.130:40011 (local address 127.0.0.1:37002)
0806 02:57:14.673236 (+ 15140us) negotiation.cc:107] Waiting for socket to connect
0806 02:57:14.673262 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 02:57:14.673407 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:57:14.674395 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:57:14.674404 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:57:14.674455 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:57:14.675173 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:14.675192 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:14.677385 (+ 2193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:14.677389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:14.680304 (+ 2915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:14.680318 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:14.681660 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:14.681666 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:14.682118 (+ 452us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:14.682188 (+ 70us) client_negotiation.cc:769] Sending connection context
0806 02:57:14.682263 (+ 75us) client_negotiation.cc:240] Negotiation successful
0806 02:57:14.684378 (+ 2115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15005,"mutex_wait_us":445,"spinlock_wait_cycles":7296,"thread_start_us":14097,"threads_started":1}
I20260806 02:57:14.687605 22354 tablet_service.cc:1467] Processing CreateTablet for tablet 5602118adf464f349deeceedb16ddc07 (DEFAULT_TABLE table=test-table [id=fd05c3db4c344285be88768ba5903123]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:57:14.687577 22555 tablet_service.cc:1467] Processing CreateTablet for tablet 5602118adf464f349deeceedb16ddc07 (DEFAULT_TABLE table=test-table [id=fd05c3db4c344285be88768ba5903123]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:57:14.687577 22160 tablet_service.cc:1467] Processing CreateTablet for tablet 5602118adf464f349deeceedb16ddc07 (DEFAULT_TABLE table=test-table [id=fd05c3db4c344285be88768ba5903123]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:57:14.689020 22160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5602118adf464f349deeceedb16ddc07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:57:14.689031 22555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5602118adf464f349deeceedb16ddc07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:57:14.689038 22354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5602118adf464f349deeceedb16ddc07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:57:14.701645 22685 tablet_bootstrap.cc:492] T 5602118adf464f349deeceedb16ddc07 P 1474992d186d4677b0ff0989920739b5: Bootstrap starting.
I20260806 02:57:14.705389 22685 tablet_bootstrap.cc:654] T 5602118adf464f349deeceedb16ddc07 P 1474992d186d4677b0ff0989920739b5: Neither blocks nor log segments found. Creating new log.
I20260806 02:57:14.706521 22685 log.cc:824] T 5602118adf464f349deeceedb16ddc07 P 1474992d186d4677b0ff0989920739b5: Log is configured to *not* fsync() on all Append() calls
I20260806 02:57:14.707371 22687 tablet_bootstrap.cc:492] T 5602118adf464f349deeceedb16ddc07 P 110d7b90e2ce4c98a94d339c8a522b89: Bootstrap starting.
I20260806 02:57:14.707832 22686 tablet_bootstrap.cc:492] T 5602118adf464f349deeceedb16ddc07 P 5b1b933eb9534a6391f00183d1f07827: Bootstrap starting.
I20260806 02:57:14.711409 22687 tablet_bootstrap.cc:654] T 5602118adf464f349deeceedb16ddc07 P 110d7b90e2ce4c98a94d339c8a522b89: Neither blocks nor log segments found. Creating new log.
I20260806 02:57:14.711447 22686 tablet_bootstrap.cc:654] T 5602118adf464f349deeceedb16ddc07 P 5b1b933eb9534a6391f00183d1f07827: Neither blocks nor log segments found. Creating new log.
I20260806 02:57:14.712491 22686 log.cc:824] T 5602118adf464f349deeceedb16ddc07 P 5b1b933eb9534a6391f00183d1f07827: Log is configured to *not* fsync() on all Append() calls
I20260806 02:57:14.712615 22687 log.cc:824] T 5602118adf464f349deeceedb16ddc07 P 110d7b90e2ce4c98a94d339c8a522b89: Log is configured to *not* fsync() on all Append() calls
I20260806 02:57:14.713255 22685 tablet_bootstrap.cc:492] T 5602118adf464f349deeceedb16ddc07 P 1474992d186d4677b0ff0989920739b5: No bootstrap required, opened a new log
I20260806 02:57:14.713559 22685 ts_tablet_manager.cc:1397] T 5602118adf464f349deeceedb16ddc07 P 1474992d186d4677b0ff0989920739b5: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20260806 02:57:14.715484 22687 tablet_bootstrap.cc:492] T 5602118adf464f349deeceedb16ddc07 P 110d7b90e2ce4c98a94d339c8a522b89: No bootstrap required, opened a new log
I20260806 02:57:14.715741 22686 tablet_bootstrap.cc:492] T 5602118adf464f349deeceedb16ddc07 P 5b1b933eb9534a6391f00183d1f07827: No bootstrap required, opened a new log
I20260806 02:57:14.715770 22687 ts_tablet_manager.cc:1397] T 5602118adf464f349deeceedb16ddc07 P 110d7b90e2ce4c98a94d339c8a522b89: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.005s
I20260806 02:57:14.716013 22686 ts_tablet_manager.cc:1397] T 5602118adf464f349deeceedb16ddc07 P 5b1b933eb9534a6391f00183d1f07827: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260806 02:57:14.723413 22687 raft_consensus.cc:348] T 5602118adf464f349deeceedb16ddc07 P 110d7b90e2ce4c98a94d339c8a522b89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110d7b90e2ce4c98a94d339c8a522b89" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 39405 } } peers { permanent_uuid: "5b1b933eb9534a6391f00183d1f07827" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 39831 } } peers { permanent_uuid: "1474992d186d4677b0ff0989920739b5" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40011 } }
I20260806 02:57:14.723872 22687 raft_consensus.cc:374] T 5602118adf464f349deeceedb16ddc07 P 110d7b90e2ce4c98a94d339c8a522b89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:57:14.723971 22687 raft_consensus.cc:729] T 5602118adf464f349deeceedb16ddc07 P 110d7b90e2ce4c98a94d339c8a522b89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 110d7b90e2ce4c98a94d339c8a522b89, State: Initialized, Role: FOLLOWER
I20260806 02:57:14.724165 22685 raft_consensus.cc:348] T 5602118adf464f349deeceedb16ddc07 P 1474992d186d4677b0ff0989920739b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110d7b90e2ce4c98a94d339c8a522b89" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 39405 } } peers { permanent_uuid: "5b1b933eb9534a6391f00183d1f07827" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 39831 } } peers { permanent_uuid: "1474992d186d4677b0ff0989920739b5" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40011 } }
I20260806 02:57:14.724670 22685 raft_consensus.cc:374] T 5602118adf464f349deeceedb16ddc07 P 1474992d186d4677b0ff0989920739b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:57:14.724778 22685 raft_consensus.cc:729] T 5602118adf464f349deeceedb16ddc07 P 1474992d186d4677b0ff0989920739b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1474992d186d4677b0ff0989920739b5, State: Initialized, Role: FOLLOWER
I20260806 02:57:14.724702 22687 consensus_queue.cc:260] T 5602118adf464f349deeceedb16ddc07 P 110d7b90e2ce4c98a94d339c8a522b89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110d7b90e2ce4c98a94d339c8a522b89" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 39405 } } peers { permanent_uuid: "5b1b933eb9534a6391f00183d1f07827" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 39831 } } peers { permanent_uuid: "1474992d186d4677b0ff0989920739b5" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40011 } }
I20260806 02:57:14.725965 22658 heartbeater.cc:503] Master 127.25.211.190:39991 was elected leader, sending a full tablet report...
I20260806 02:57:14.725449 22685 consensus_queue.cc:260] T 5602118adf464f349deeceedb16ddc07 P 1474992d186d4677b0ff0989920739b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110d7b90e2ce4c98a94d339c8a522b89" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 39405 } } peers { permanent_uuid: "5b1b933eb9534a6391f00183d1f07827" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 39831 } } peers { permanent_uuid: "1474992d186d4677b0ff0989920739b5" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40011 } }
I20260806 02:57:14.726742 22687 ts_tablet_manager.cc:1428] T 5602118adf464f349deeceedb16ddc07 P 110d7b90e2ce4c98a94d339c8a522b89: Time spent starting tablet: real 0.011s user 0.007s sys 0.005s
I20260806 02:57:14.729355 22686 raft_consensus.cc:348] T 5602118adf464f349deeceedb16ddc07 P 5b1b933eb9534a6391f00183d1f07827 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110d7b90e2ce4c98a94d339c8a522b89" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 39405 } } peers { permanent_uuid: "5b1b933eb9534a6391f00183d1f07827" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 39831 } } peers { permanent_uuid: "1474992d186d4677b0ff0989920739b5" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40011 } }
I20260806 02:57:14.729923 22686 raft_consensus.cc:374] T 5602118adf464f349deeceedb16ddc07 P 5b1b933eb9534a6391f00183d1f07827 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:57:14.730033 22686 raft_consensus.cc:729] T 5602118adf464f349deeceedb16ddc07 P 5b1b933eb9534a6391f00183d1f07827 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b1b933eb9534a6391f00183d1f07827, State: Initialized, Role: FOLLOWER
I20260806 02:57:14.730930 22450 heartbeater.cc:503] Master 127.25.211.190:39991 was elected leader, sending a full tablet report...
I20260806 02:57:14.730832 22686 consensus_queue.cc:260] T 5602118adf464f349deeceedb16ddc07 P 5b1b933eb9534a6391f00183d1f07827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110d7b90e2ce4c98a94d339c8a522b89" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 39405 } } peers { permanent_uuid: "5b1b933eb9534a6391f00183d1f07827" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 39831 } } peers { permanent_uuid: "1474992d186d4677b0ff0989920739b5" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40011 } }
I20260806 02:57:14.733034 22686 ts_tablet_manager.cc:1428] T 5602118adf464f349deeceedb16ddc07 P 5b1b933eb9534a6391f00183d1f07827: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s
I20260806 02:57:14.735067 22685 ts_tablet_manager.cc:1428] T 5602118adf464f349deeceedb16ddc07 P 1474992d186d4677b0ff0989920739b5: Time spent starting tablet: real 0.021s user 0.008s sys 0.008s
I20260806 02:57:14.838851 22698 raft_consensus.cc:482] T 5602118adf464f349deeceedb16ddc07 P 5b1b933eb9534a6391f00183d1f07827 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:57:14.839169 22698 raft_consensus.cc:504] T 5602118adf464f349deeceedb16ddc07 P 5b1b933eb9534a6391f00183d1f07827 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110d7b90e2ce4c98a94d339c8a522b89" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 39405 } } peers { permanent_uuid: "5b1b933eb9534a6391f00183d1f07827" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 39831 } } peers { permanent_uuid: "1474992d186d4677b0ff0989920739b5" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40011 } }
I20260806 02:57:14.840881 22698 leader_election.cc:288] T 5602118adf464f349deeceedb16ddc07 P 5b1b933eb9534a6391f00183d1f07827 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 110d7b90e2ce4c98a94d339c8a522b89 (127.25.211.131:39405), 1474992d186d4677b0ff0989920739b5 (127.25.211.130:40011)
I20260806 02:57:14.847908 22683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:14.842453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:55527 (local address 127.25.211.130:40011)
0806 02:57:14.842659 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:14.842665 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:14.842929 (+ 264us) server_negotiation.cc:407] Connection header received
0806 02:57:14.843112 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:14.843116 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:14.843185 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:14.843262 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:57:14.843836 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:14.844496 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:14.846307 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:14.847040 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:14.847174 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:14.847251 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:14.847531 (+ 280us) server_negotiation.cc:299] Negotiation successful
0806 02:57:14.847652 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 02:57:14.848007 22700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:14.842264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.130:40011 (local address 127.25.211.129:55527)
0806 02:57:14.842835 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0806 02:57:14.842849 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 02:57:14.842989 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:57:14.843304 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:57:14.843308 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:57:14.843332 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:57:14.843687 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:14.843696 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:14.844633 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:14.844637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:14.846114 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:14.846126 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:14.847221 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:14.847228 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:14.847394 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:14.847416 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 02:57:14.847490 (+ 74us) client_negotiation.cc:240] Negotiation successful
0806 02:57:14.847608 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"mutex_wait_us":27,"thread_start_us":396,"threads_started":1}
I20260806 02:57:14.848042 22699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:14.841305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.131:39405 (local address 127.25.211.129:37193)
0806 02:57:14.842042 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0806 02:57:14.842062 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 02:57:14.842250 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:57:14.842736 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:57:14.842741 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:57:14.842783 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:57:14.843144 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:14.843154 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:14.844275 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:14.844279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:14.845927 (+ 1648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:14.845941 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:14.847098 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:14.847104 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:14.847294 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:14.847318 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 02:57:14.847401 (+ 83us) client_negotiation.cc:240] Negotiation successful
0806 02:57:14.847537 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"mutex_wait_us":108,"spinlock_wait_cycles":1280,"thread_start_us":413,"threads_started":1}
W20260806 02:57:14.848083 22659 tablet.cc:2369] T 5602118adf464f349deeceedb16ddc07 P 110d7b90e2ce4c98a94d339c8a522b89: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:57:14.848443 22677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:14.841396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:37193 (local address 127.25.211.131:39405)
0806 02:57:14.841649 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:14.841655 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:14.842186 (+ 531us) server_negotiation.cc:407] Connection header received
0806 02:57:14.842479 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:14.842485 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:14.842572 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:14.842700 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:57:14.843287 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:14.844148 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:14.846124 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:14.846916 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:14.847078 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:14.847148 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:14.847442 (+ 294us) server_negotiation.cc:299] Negotiation successful
0806 02:57:14.847564 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260806 02:57:14.849110 22379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5602118adf464f349deeceedb16ddc07" candidate_uuid: "5b1b933eb9534a6391f00183d1f07827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1474992d186d4677b0ff0989920739b5" is_pre_election: true
I20260806 02:57:14.849291 22586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5602118adf464f349deeceedb16ddc07" candidate_uuid: "5b1b933eb9534a6391f00183d1f07827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "110d7b90e2ce4c98a94d339c8a522b89" is_pre_election: true
I20260806 02:57:14.849622 22379 raft_consensus.cc:2454] T 5602118adf464f349deeceedb16ddc07 P 1474992d186d4677b0ff0989920739b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b1b933eb9534a6391f00183d1f07827 in term 0.
I20260806 02:57:14.849772 22586 raft_consensus.cc:2454] T 5602118adf464f349deeceedb16ddc07 P 110d7b90e2ce4c98a94d339c8a522b89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b1b933eb9534a6391f00183d1f07827 in term 0.
I20260806 02:57:14.850281 22094 leader_election.cc:302] T 5602118adf464f349deeceedb16ddc07 P 5b1b933eb9534a6391f00183d1f07827 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1474992d186d4677b0ff0989920739b5, 5b1b933eb9534a6391f00183d1f07827; no voters:
I20260806 02:57:14.850643 22698 raft_consensus.cc:2781] T 5602118adf464f349deeceedb16ddc07 P 5b1b933eb9534a6391f00183d1f07827 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:57:14.850812 22698 raft_consensus.cc:482] T 5602118adf464f349deeceedb16ddc07 P 5b1b933eb9534a6391f00183d1f07827 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:57:14.850922 22698 raft_consensus.cc:3037] T 5602118adf464f349deeceedb16ddc07 P 5b1b933eb9534a6391f00183d1f07827 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:57:14.853080 22698 raft_consensus.cc:504] T 5602118adf464f349deeceedb16ddc07 P 5b1b933eb9534a6391f00183d1f07827 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110d7b90e2ce4c98a94d339c8a522b89" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 39405 } } peers { permanent_uuid: "5b1b933eb9534a6391f00183d1f07827" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 39831 } } peers { permanent_uuid: "1474992d186d4677b0ff0989920739b5" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40011 } }
I20260806 02:57:14.853629 22698 leader_election.cc:288] T 5602118adf464f349deeceedb16ddc07 P 5b1b933eb9534a6391f00183d1f07827 [CANDIDATE]: Term 1 election: Requested vote from peers 110d7b90e2ce4c98a94d339c8a522b89 (127.25.211.131:39405), 1474992d186d4677b0ff0989920739b5 (127.25.211.130:40011)
I20260806 02:57:14.854043 22586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5602118adf464f349deeceedb16ddc07" candidate_uuid: "5b1b933eb9534a6391f00183d1f07827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "110d7b90e2ce4c98a94d339c8a522b89"
I20260806 02:57:14.854099 22379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5602118adf464f349deeceedb16ddc07" candidate_uuid: "5b1b933eb9534a6391f00183d1f07827" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1474992d186d4677b0ff0989920739b5"
I20260806 02:57:14.854259 22586 raft_consensus.cc:3037] T 5602118adf464f349deeceedb16ddc07 P 110d7b90e2ce4c98a94d339c8a522b89 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:57:14.854316 22379 raft_consensus.cc:3037] T 5602118adf464f349deeceedb16ddc07 P 1474992d186d4677b0ff0989920739b5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:57:14.856369 22586 raft_consensus.cc:2454] T 5602118adf464f349deeceedb16ddc07 P 110d7b90e2ce4c98a94d339c8a522b89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b1b933eb9534a6391f00183d1f07827 in term 1.
I20260806 02:57:14.856448 22379 raft_consensus.cc:2454] T 5602118adf464f349deeceedb16ddc07 P 1474992d186d4677b0ff0989920739b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b1b933eb9534a6391f00183d1f07827 in term 1.
I20260806 02:57:14.856709 22094 leader_election.cc:302] T 5602118adf464f349deeceedb16ddc07 P 5b1b933eb9534a6391f00183d1f07827 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 110d7b90e2ce4c98a94d339c8a522b89, 5b1b933eb9534a6391f00183d1f07827; no voters:
I20260806 02:57:14.856976 22698 raft_consensus.cc:2781] T 5602118adf464f349deeceedb16ddc07 P 5b1b933eb9534a6391f00183d1f07827 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:57:14.857390 22698 raft_consensus.cc:686] T 5602118adf464f349deeceedb16ddc07 P 5b1b933eb9534a6391f00183d1f07827 [term 1 LEADER]: Becoming Leader. State: Replica: 5b1b933eb9534a6391f00183d1f07827, State: Running, Role: LEADER
I20260806 02:57:14.857919 22698 consensus_queue.cc:237] T 5602118adf464f349deeceedb16ddc07 P 5b1b933eb9534a6391f00183d1f07827 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110d7b90e2ce4c98a94d339c8a522b89" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 39405 } } peers { permanent_uuid: "5b1b933eb9534a6391f00183d1f07827" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 39831 } } peers { permanent_uuid: "1474992d186d4677b0ff0989920739b5" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40011 } }
I20260806 02:57:14.862563 21991 catalog_manager.cc:5495] T 5602118adf464f349deeceedb16ddc07 P 5b1b933eb9534a6391f00183d1f07827 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b1b933eb9534a6391f00183d1f07827 (127.25.211.129). New cstate: current_term: 1 leader_uuid: "5b1b933eb9534a6391f00183d1f07827" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "110d7b90e2ce4c98a94d339c8a522b89" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 39405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b1b933eb9534a6391f00183d1f07827" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 39831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1474992d186d4677b0ff0989920739b5" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40011 } health_report { overall_health: UNKNOWN } } }
W20260806 02:57:14.873205 22451 tablet.cc:2369] T 5602118adf464f349deeceedb16ddc07 P 1474992d186d4677b0ff0989920739b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:57:14.882500 22680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:14.877090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50876 (local address 127.25.211.129:39831)
0806 02:57:14.877291 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:14.877297 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:14.877317 (+ 20us) server_negotiation.cc:407] Connection header received
0806 02:57:14.877460 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:14.877464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:14.877575 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:14.877702 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:57:14.878096 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:14.878827 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:14.880211 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:14.880632 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:14.880753 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:14.881362 (+ 609us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:57:14.881862 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:57:14.882038 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:14.882236 (+ 198us) server_negotiation.cc:299] Negotiation successful
0806 02:57:14.882287 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 02:57:14.890647 22379 raft_consensus.cc:1261] T 5602118adf464f349deeceedb16ddc07 P 1474992d186d4677b0ff0989920739b5 [term 1 FOLLOWER]: Refusing update from remote peer 5b1b933eb9534a6391f00183d1f07827: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:57:14.890739 22586 raft_consensus.cc:1261] T 5602118adf464f349deeceedb16ddc07 P 110d7b90e2ce4c98a94d339c8a522b89 [term 1 FOLLOWER]: Refusing update from remote peer 5b1b933eb9534a6391f00183d1f07827: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:57:14.891585 22701 consensus_queue.cc:1035] T 5602118adf464f349deeceedb16ddc07 P 5b1b933eb9534a6391f00183d1f07827 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1474992d186d4677b0ff0989920739b5" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 40011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:57:14.891957 22698 consensus_queue.cc:1035] T 5602118adf464f349deeceedb16ddc07 P 5b1b933eb9534a6391f00183d1f07827 [LEADER]: Connected to new peer: Peer: permanent_uuid: "110d7b90e2ce4c98a94d339c8a522b89" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 39405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 02:57:14.901708 22262 tablet.cc:2369] T 5602118adf464f349deeceedb16ddc07 P 5b1b933eb9534a6391f00183d1f07827: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:57:14.904117 22708 mvcc.cc:205] Tried to move back new op lower bound from 7315394702902448128 to 7315394702781865984. Current Snapshot: MvccSnapshot[applied={T|T < 7315394702902448128}]
I20260806 02:57:14.908408 22705 mvcc.cc:205] Tried to move back new op lower bound from 7315394702902448128 to 7315394702781865984. Current Snapshot: MvccSnapshot[applied={T|T < 7315394702902448128 or (T in {7315394702902448128})}]
I20260806 02:57:14.917650 21988 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:57:14.917906 21988 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:57:14.921650 21988 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 09bf27ac30694e14884d890e2381a329: Sending DeleteTablet for 3 replicas of tablet 5602118adf464f349deeceedb16ddc07
I20260806 02:57:14.922922 22354 tablet_service.cc:1514] Processing DeleteTablet for tablet 5602118adf464f349deeceedb16ddc07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:57:14 UTC) from {username='slave'} at 127.0.0.1:37002
I20260806 02:57:14.922917 22160 tablet_service.cc:1514] Processing DeleteTablet for tablet 5602118adf464f349deeceedb16ddc07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:57:14 UTC) from {username='slave'} at 127.0.0.1:50860
I20260806 02:57:14.923044 22555 tablet_service.cc:1514] Processing DeleteTablet for tablet 5602118adf464f349deeceedb16ddc07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:57:14 UTC) from {username='slave'} at 127.0.0.1:42896
I20260806 02:57:14.927258 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 22075
I20260806 02:57:14.941075 22717 tablet_replica.cc:330] T 5602118adf464f349deeceedb16ddc07 P 5b1b933eb9534a6391f00183d1f07827: stopping tablet replica
I20260806 02:57:14.941787 22717 raft_consensus.cc:2229] T 5602118adf464f349deeceedb16ddc07 P 5b1b933eb9534a6391f00183d1f07827 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:57:14.942780 22717 raft_consensus.cc:2258] T 5602118adf464f349deeceedb16ddc07 P 5b1b933eb9534a6391f00183d1f07827 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:57:14.943787 22719 tablet_replica.cc:330] T 5602118adf464f349deeceedb16ddc07 P 110d7b90e2ce4c98a94d339c8a522b89: stopping tablet replica
I20260806 02:57:14.943962 22718 tablet_replica.cc:330] T 5602118adf464f349deeceedb16ddc07 P 1474992d186d4677b0ff0989920739b5: stopping tablet replica
I20260806 02:57:14.944339 22719 raft_consensus.cc:2229] T 5602118adf464f349deeceedb16ddc07 P 110d7b90e2ce4c98a94d339c8a522b89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:57:14.944429 22718 raft_consensus.cc:2229] T 5602118adf464f349deeceedb16ddc07 P 1474992d186d4677b0ff0989920739b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:57:14.945024 22718 raft_consensus.cc:2258] T 5602118adf464f349deeceedb16ddc07 P 1474992d186d4677b0ff0989920739b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:57:14.945024 22719 raft_consensus.cc:2258] T 5602118adf464f349deeceedb16ddc07 P 110d7b90e2ce4c98a94d339c8a522b89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:57:14.946555 22717 ts_tablet_manager.cc:1905] T 5602118adf464f349deeceedb16ddc07 P 5b1b933eb9534a6391f00183d1f07827: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:57:14.947161 22718 ts_tablet_manager.cc:1905] T 5602118adf464f349deeceedb16ddc07 P 1474992d186d4677b0ff0989920739b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:57:14.947168 22719 ts_tablet_manager.cc:1905] T 5602118adf464f349deeceedb16ddc07 P 110d7b90e2ce4c98a94d339c8a522b89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:57:14.951117 22680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:14.928612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50888 (local address 127.25.211.129:39831)
0806 02:57:14.928790 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:14.928797 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:14.931730 (+ 2933us) server_negotiation.cc:407] Connection header received
0806 02:57:14.932592 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:14.932599 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:14.932722 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:14.932796 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:14.934090 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:14.935409 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:14.939752 (+ 4343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:14.940520 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:14.940627 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:14.943196 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:14.943208 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:14.943228 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:57:14.944092 (+ 864us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:14.944170 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:57:14.944196 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:14.944284 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:14.950598 (+ 6314us) server_negotiation.cc:299] Negotiation successful
0806 02:57:14.950725 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 02:57:14.952406 22254 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:57:14.954252 22718 ts_tablet_manager.cc:1918] T 5602118adf464f349deeceedb16ddc07 P 1474992d186d4677b0ff0989920739b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:57:14.954526 22718 log.cc:1196] T 5602118adf464f349deeceedb16ddc07 P 1474992d186d4677b0ff0989920739b5: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/ts-1/wal/wals/5602118adf464f349deeceedb16ddc07
I20260806 02:57:14.955183 22718 ts_tablet_manager.cc:1939] T 5602118adf464f349deeceedb16ddc07 P 1474992d186d4677b0ff0989920739b5: Deleting consensus metadata
I20260806 02:57:14.955971 22719 ts_tablet_manager.cc:1918] T 5602118adf464f349deeceedb16ddc07 P 110d7b90e2ce4c98a94d339c8a522b89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:57:14.956168 22719 log.cc:1196] T 5602118adf464f349deeceedb16ddc07 P 110d7b90e2ce4c98a94d339c8a522b89: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1785984707115702-26446-0/minicluster-data/ts-2/wal/wals/5602118adf464f349deeceedb16ddc07
I20260806 02:57:14.956745 22719 ts_tablet_manager.cc:1939] T 5602118adf464f349deeceedb16ddc07 P 110d7b90e2ce4c98a94d339c8a522b89: Deleting consensus metadata
I20260806 02:57:14.957553 21968 catalog_manager.cc:4841] TS 1474992d186d4677b0ff0989920739b5 (127.25.211.130:40011): tablet 5602118adf464f349deeceedb16ddc07 (table test-table [id=fd05c3db4c344285be88768ba5903123]) successfully deleted
I20260806 02:57:14.958410 21968 catalog_manager.cc:4841] TS 110d7b90e2ce4c98a94d339c8a522b89 (127.25.211.131:39405): tablet 5602118adf464f349deeceedb16ddc07 (table test-table [id=fd05c3db4c344285be88768ba5903123]) successfully deleted
I20260806 02:57:15.056559 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 22075
I20260806 02:57:15.078908 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 22265
W20260806 02:57:15.078904 21972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.25.211.129:39831 (error 108)
I20260806 02:57:15.081358 22678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:15.080807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:39831 (local address 127.0.0.1:50890)
0806 02:57:15.080997 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0806 02:57:15.081128 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.129:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260806 02:57:15.081653 21972 catalog_manager.cc:4584] TS 5b1b933eb9534a6391f00183d1f07827 (127.25.211.129:39831): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5602118adf464f349deeceedb16ddc07: Network error: Client connection negotiation failed: client connection to 127.25.211.129:39831: connect: Connection refused (error 111)
I20260806 02:57:15.086615 22683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:15.079818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37014 (local address 127.25.211.130:40011)
0806 02:57:15.080006 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:15.080012 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:15.080033 (+ 21us) server_negotiation.cc:407] Connection header received
0806 02:57:15.080234 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:15.080238 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:15.080319 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:15.080420 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:15.080956 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:15.081928 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:15.083323 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:15.083751 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:15.083868 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:15.085053 (+ 1185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:15.085064 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:15.085080 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:57:15.085708 (+ 628us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:15.085764 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:57:15.085787 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:15.085891 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:15.086184 (+ 293us) server_negotiation.cc:299] Negotiation successful
0806 02:57:15.086315 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 02:57:15.087361 22442 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:57:15.135542 22678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:15.134927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:39831 (local address 127.0.0.1:50902)
0806 02:57:15.135221 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0806 02:57:15.135334 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.129:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152}
I20260806 02:57:15.136734 22678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:15.136323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:39831 (local address 127.0.0.1:50914)
0806 02:57:15.136483 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0806 02:57:15.136589 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.129:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260806 02:57:15.176904 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 22265
I20260806 02:57:15.197463 22678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:15.196970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:39831 (local address 127.0.0.1:50916)
0806 02:57:15.197194 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0806 02:57:15.197284 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.129:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260806 02:57:15.198491 22678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:15.198139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:39831 (local address 127.0.0.1:50928)
0806 02:57:15.198299 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0806 02:57:15.198367 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.129:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260806 02:57:15.198742 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 22455
I20260806 02:57:15.206115 22677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:15.199547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42898 (local address 127.25.211.131:39405)
0806 02:57:15.199736 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:15.199742 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:15.199764 (+ 22us) server_negotiation.cc:407] Connection header received
0806 02:57:15.199878 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:15.199882 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:15.199968 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:15.200060 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:15.200558 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:15.201639 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:15.203109 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:15.203527 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:15.203653 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:15.204752 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:15.204762 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:15.204785 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:57:15.205366 (+ 581us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:15.205416 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:57:15.205437 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:15.205558 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:15.205687 (+ 129us) server_negotiation.cc:299] Negotiation successful
0806 02:57:15.205808 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 02:57:15.206780 22653 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:57:15.279480 22678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:15.278883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:39831 (local address 127.0.0.1:50932)
0806 02:57:15.279135 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0806 02:57:15.279253 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.129:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260806 02:57:15.280716 22678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:15.280236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:39831 (local address 127.0.0.1:50934)
0806 02:57:15.280462 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0806 02:57:15.280554 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.129:39831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260806 02:57:15.296607 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 22455
I20260806 02:57:15.318373 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 21944
I20260806 02:57:15.319511 22035 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:57:15.429653 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 21944
2026-08-06T02:57:15Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3866 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:57:15 dist-test-slave-zj8q krb5kdc[22729](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:57:15 dist-test-slave-zj8q krb5kdc[22729](info): set up 2 sockets
Aug 06 02:57:15 dist-test-slave-zj8q krb5kdc[22729](info): commencing operation
krb5kdc: starting...
W20260806 02:57:16.674603 26446 mini_kdc.cc:121] Time spent starting KDC: real 1.196s user 0.006s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Aug 06 02:57:16 dist-test-slave-zj8q krb5kdc[22729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:57:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:57:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.211.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab.
Entry for principal kudu/127.25.211.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab.
I20260806 02:57:17.013733 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:42401
--webserver_interface=127.25.211.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:41907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.211.190:42401
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:57:17.368686 22750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:57:17.369016 22750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:57:17.369139 22750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:57:17.376709 22750 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:57:17.376828 22750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:57:17.376888 22750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:57:17.376971 22750 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:57:17.377022 22750 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:57:17.387606 22750 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:41907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.211.190:42401
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:42401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.211.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:57:17.389376 22750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:57:17.391027 22750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:57:17.403227 22760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:17.403501 22758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:17.403940 22757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:57:17.404628 22750 server_base.cc:1034] running on GCE node
I20260806 02:57:17.405475 22750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:57:17.407085 22750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:57:17.408325 22750 hybrid_clock.cc:648] HybridClock initialized: now 1785985037408265 us; error 53 us; skew 500 ppm
I20260806 02:57:17.408700 22750 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:57:17 dist-test-slave-zj8q krb5kdc[22729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985037, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:57:17.411959 22750 init.cc:377] Logged in from keytab as kudu/127.25.211.190@KRBTEST.COM (short username kudu)
I20260806 02:57:17.413415 22750 webserver.cc:460] Webserver started at http://127.25.211.190:44049/ using document root <none> and password file <none>
I20260806 02:57:17.414088 22750 fs_manager.cc:359] Metadata directory not provided
I20260806 02:57:17.414229 22750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:57:17.414501 22750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:57:17.417405 22750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/master-0/data/instance:
uuid: "37eb4d9ea59d4ebdb00aeffbdd541dcb"
format_stamp: "Formatted at 2026-08-06 02:57:17 on dist-test-slave-zj8q"
server_key: "5867881fcbaab2b4019f2890ea5f7a8a"
server_key_iv: "683354a255dbc03a52b9d05643992fab"
server_key_version: "encryptionkey@0"
I20260806 02:57:17.418102 22750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/master-0/wal/instance:
uuid: "37eb4d9ea59d4ebdb00aeffbdd541dcb"
format_stamp: "Formatted at 2026-08-06 02:57:17 on dist-test-slave-zj8q"
server_key: "5867881fcbaab2b4019f2890ea5f7a8a"
server_key_iv: "683354a255dbc03a52b9d05643992fab"
server_key_version: "encryptionkey@0"
I20260806 02:57:17.423266 22750 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 02:57:17.426702 22769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:17.428025 22750 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260806 02:57:17.428201 22750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/master-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/master-0/wal
uuid: "37eb4d9ea59d4ebdb00aeffbdd541dcb"
format_stamp: "Formatted at 2026-08-06 02:57:17 on dist-test-slave-zj8q"
server_key: "5867881fcbaab2b4019f2890ea5f7a8a"
server_key_iv: "683354a255dbc03a52b9d05643992fab"
server_key_version: "encryptionkey@0"
I20260806 02:57:17.428385 22750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:57:17.450382 22750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:57:17.454916 22750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:57:17.455227 22750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:57:17.475819 22750 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.190:42401
I20260806 02:57:17.475844 22837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.190:42401 every 8 connection(s)
I20260806 02:57:17.477243 22750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
I20260806 02:57:17.478397 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 22750
I20260806 02:57:17.478626 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/master-0/wal/instance
I20260806 02:57:17.479166 26446 external_mini_cluster.cc:1428] Setting key 724da235e180989e2bb502bac07550a0
I20260806 02:57:17.482228 22838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 06 02:57:17 dist-test-slave-zj8q krb5kdc[22729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:57:17.496043 22838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37eb4d9ea59d4ebdb00aeffbdd541dcb: Bootstrap starting.
I20260806 02:57:17.499454 22838 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37eb4d9ea59d4ebdb00aeffbdd541dcb: Neither blocks nor log segments found. Creating new log.
I20260806 02:57:17.500752 22838 log.cc:824] T 00000000000000000000000000000000 P 37eb4d9ea59d4ebdb00aeffbdd541dcb: Log is configured to *not* fsync() on all Append() calls
I20260806 02:57:17.503710 22843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:17.481303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56946 (local address 127.25.211.190:42401)
0806 02:57:17.482141 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:17.482179 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:17.482234 (+ 55us) server_negotiation.cc:407] Connection header received
0806 02:57:17.483975 (+ 1741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:17.483998 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:17.484380 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:17.484734 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:17.486655 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:17.488066 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:17.489942 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:17.490702 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:17.490847 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:17.495971 (+ 5124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:17.495980 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:17.495999 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:57:17.496050 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:17.499104 (+ 3054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:17.500082 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:17.500090 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:17.500097 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:17.500301 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:17.500782 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:17.500785 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:17.500788 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:17.501511 (+ 723us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:17.501846 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:17.502311 (+ 465us) server_negotiation.cc:299] Negotiation successful
0806 02:57:17.502738 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":390,"threads_started":1}
I20260806 02:57:17.504048 22838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37eb4d9ea59d4ebdb00aeffbdd541dcb: No bootstrap required, opened a new log
I20260806 02:57:17.511884 22838 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37eb4d9ea59d4ebdb00aeffbdd541dcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37eb4d9ea59d4ebdb00aeffbdd541dcb" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 42401 } }
I20260806 02:57:17.512347 22838 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37eb4d9ea59d4ebdb00aeffbdd541dcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:57:17.512449 22838 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37eb4d9ea59d4ebdb00aeffbdd541dcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37eb4d9ea59d4ebdb00aeffbdd541dcb, State: Initialized, Role: FOLLOWER
I20260806 02:57:17.513039 22838 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37eb4d9ea59d4ebdb00aeffbdd541dcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37eb4d9ea59d4ebdb00aeffbdd541dcb" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 42401 } }
I20260806 02:57:17.513293 22838 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37eb4d9ea59d4ebdb00aeffbdd541dcb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:57:17.513430 22838 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37eb4d9ea59d4ebdb00aeffbdd541dcb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:57:17.513600 22838 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37eb4d9ea59d4ebdb00aeffbdd541dcb [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:57:17.516000 22838 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37eb4d9ea59d4ebdb00aeffbdd541dcb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37eb4d9ea59d4ebdb00aeffbdd541dcb" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 42401 } }
I20260806 02:57:17.516544 22838 leader_election.cc:302] T 00000000000000000000000000000000 P 37eb4d9ea59d4ebdb00aeffbdd541dcb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37eb4d9ea59d4ebdb00aeffbdd541dcb; no voters:
I20260806 02:57:17.517165 22838 leader_election.cc:288] T 00000000000000000000000000000000 P 37eb4d9ea59d4ebdb00aeffbdd541dcb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:57:17.517290 22846 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37eb4d9ea59d4ebdb00aeffbdd541dcb [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:57:17.517948 22846 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37eb4d9ea59d4ebdb00aeffbdd541dcb [term 1 LEADER]: Becoming Leader. State: Replica: 37eb4d9ea59d4ebdb00aeffbdd541dcb, State: Running, Role: LEADER
I20260806 02:57:17.518548 22846 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37eb4d9ea59d4ebdb00aeffbdd541dcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37eb4d9ea59d4ebdb00aeffbdd541dcb" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 42401 } }
I20260806 02:57:17.519361 22838 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37eb4d9ea59d4ebdb00aeffbdd541dcb [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:57:17.521800 22847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37eb4d9ea59d4ebdb00aeffbdd541dcb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37eb4d9ea59d4ebdb00aeffbdd541dcb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37eb4d9ea59d4ebdb00aeffbdd541dcb" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 42401 } } }
I20260806 02:57:17.522280 22847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37eb4d9ea59d4ebdb00aeffbdd541dcb [sys.catalog]: This master's current role is: LEADER
I20260806 02:57:17.521839 22848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37eb4d9ea59d4ebdb00aeffbdd541dcb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37eb4d9ea59d4ebdb00aeffbdd541dcb. Latest consensus state: current_term: 1 leader_uuid: "37eb4d9ea59d4ebdb00aeffbdd541dcb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37eb4d9ea59d4ebdb00aeffbdd541dcb" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 42401 } } }
I20260806 02:57:17.525870 22848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37eb4d9ea59d4ebdb00aeffbdd541dcb [sys.catalog]: This master's current role is: LEADER
I20260806 02:57:17.526923 22854 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:57:17.531442 22854 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:57:17.539829 22854 catalog_manager.cc:1347] Generated new cluster ID: b6e810090d904dd49f60376c542fb680
I20260806 02:57:17.539964 22854 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:57:17.548574 22854 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:57:17.550541 22854 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:57:17.569234 22854 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37eb4d9ea59d4ebdb00aeffbdd541dcb: Generated new TSK 0
I20260806 02:57:17.570442 22854 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.25.211.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab.
Entry for principal kudu/127.25.211.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab.
I20260806 02:57:17.661649 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.129:0
--local_ip_for_outbound_sockets=127.25.211.129
--webserver_interface=127.25.211.129
--webserver_port=0
--tserver_master_addrs=127.25.211.190:42401
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab
--principal=kudu/127.25.211.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:41907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:57:18.010213 22883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:57:18.010529 22883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:57:18.010677 22883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:57:18.020828 22883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:57:18.020963 22883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:57:18.021155 22883 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.129
I20260806 02:57:18.032650 22883 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:41907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.129.keytab
--principal=kudu/127.25.211.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.211.129
--webserver_port=0
--tserver_master_addrs=127.25.211.190:42401
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.129
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:57:18.034124 22883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:57:18.035636 22883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:57:18.050134 22890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:18.050201 22888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:18.050197 22893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:57:18.051015 22883 server_base.cc:1034] running on GCE node
I20260806 02:57:18.051693 22883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:57:18.052790 22883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:57:18.054032 22883 hybrid_clock.cc:648] HybridClock initialized: now 1785985038053971 us; error 50 us; skew 500 ppm
I20260806 02:57:18.054422 22883 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:57:18 dist-test-slave-zj8q krb5kdc[22729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985038, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:57:18.061851 22883 init.cc:377] Logged in from keytab as kudu/127.25.211.129@KRBTEST.COM (short username kudu)
I20260806 02:57:18.063656 22883 webserver.cc:460] Webserver started at http://127.25.211.129:43347/ using document root <none> and password file <none>
I20260806 02:57:18.064383 22883 fs_manager.cc:359] Metadata directory not provided
I20260806 02:57:18.064512 22883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:57:18.064889 22883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:57:18.067975 22883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/ts-0/data/instance:
uuid: "50eed71be1674a23b534853396f228b1"
format_stamp: "Formatted at 2026-08-06 02:57:18 on dist-test-slave-zj8q"
server_key: "34acdcaf5613e8fd20e36abd6a75d73c"
server_key_iv: "009bb1d55207dd382d55345e4b735497"
server_key_version: "encryptionkey@0"
I20260806 02:57:18.068833 22883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/ts-0/wal/instance:
uuid: "50eed71be1674a23b534853396f228b1"
format_stamp: "Formatted at 2026-08-06 02:57:18 on dist-test-slave-zj8q"
server_key: "34acdcaf5613e8fd20e36abd6a75d73c"
server_key_iv: "009bb1d55207dd382d55345e4b735497"
server_key_version: "encryptionkey@0"
I20260806 02:57:18.074291 22883 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260806 02:57:18.078032 22906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:18.079428 22883 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:57:18.079620 22883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/ts-0/wal
uuid: "50eed71be1674a23b534853396f228b1"
format_stamp: "Formatted at 2026-08-06 02:57:18 on dist-test-slave-zj8q"
server_key: "34acdcaf5613e8fd20e36abd6a75d73c"
server_key_iv: "009bb1d55207dd382d55345e4b735497"
server_key_version: "encryptionkey@0"
I20260806 02:57:18.079825 22883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:57:18.094877 22883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:57:18.099120 22883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:57:18.099412 22883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:57:18.100415 22883 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:57:18.101788 22883 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:57:18.101869 22883 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:18.102003 22883 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:57:18.102077 22883 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:18.133808 22883 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.129:44779
I20260806 02:57:18.133867 23072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.129:44779 every 8 connection(s)
I20260806 02:57:18.135238 22883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
I20260806 02:57:18.136399 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 22883
I20260806 02:57:18.136595 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/ts-0/wal/instance
I20260806 02:57:18.137156 26446 external_mini_cluster.cc:1428] Setting key 1e86f6857c39c2d70ac94097405ffd16
Aug 06 02:57:18 dist-test-slave-zj8q krb5kdc[22729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985038, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.129@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:57:18.159250 23075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:18.138330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:60801 (local address 127.25.211.190:42401)
0806 02:57:18.138866 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:18.138873 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:18.140011 (+ 1138us) server_negotiation.cc:407] Connection header received
0806 02:57:18.142582 (+ 2571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:18.142589 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:18.142669 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:18.142770 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:18.144891 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:18.145696 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:18.147906 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:18.148348 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:18.148462 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:18.152796 (+ 4334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:18.152801 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:18.152805 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:57:18.152845 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:18.155851 (+ 3006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:18.156661 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:18.156666 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:18.156668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:18.156803 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:18.157278 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:18.157282 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:18.157284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:18.157578 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:18.157719 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:18.158594 (+ 875us) server_negotiation.cc:299] Negotiation successful
0806 02:57:18.158857 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":240,"threads_started":1}
I20260806 02:57:18.160254 23076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:18.138911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:42401 (local address 127.25.211.129:60801)
0806 02:57:18.139727 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0806 02:57:18.139876 (+ 149us) client_negotiation.cc:174] Beginning negotiation
0806 02:57:18.142205 (+ 2329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:57:18.143067 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:57:18.143079 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:57:18.143839 (+ 760us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:57:18.144673 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:18.144699 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:18.145875 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:18.145880 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:18.147732 (+ 1852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:18.147749 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:18.148597 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:18.148603 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:18.148909 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:18.149741 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:57:18.149772 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:57:18.152483 (+ 2711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:57:18.156012 (+ 3529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:57:18.156023 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:57:18.156043 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:57:18.156513 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:57:18.156918 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:57:18.156923 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:57:18.156927 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:57:18.157150 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:57:18.157718 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:57:18.157727 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:57:18.158110 (+ 383us) client_negotiation.cc:769] Sending connection context
0806 02:57:18.158442 (+ 332us) client_negotiation.cc:240] Negotiation successful
0806 02:57:18.159053 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":311,"threads_started":1}
I20260806 02:57:18.161968 23073 heartbeater.cc:347] Connected to a master server at 127.25.211.190:42401
I20260806 02:57:18.162367 23073 heartbeater.cc:464] Registering TS with master...
I20260806 02:57:18.163347 23073 heartbeater.cc:511] Master 127.25.211.190:42401 requested a full tablet report, sending...
I20260806 02:57:18.166000 22795 ts_manager.cc:194] Registered new tserver with Master: 50eed71be1674a23b534853396f228b1 (127.25.211.129:44779)
I20260806 02:57:18.168103 22795 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.211.129@KRBTEST.COM'} at 127.25.211.129:60801
WARNING: no policy specified for kudu/127.25.211.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab.
Entry for principal kudu/127.25.211.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab.
I20260806 02:57:18.213400 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.130:0
--local_ip_for_outbound_sockets=127.25.211.130
--webserver_interface=127.25.211.130
--webserver_port=0
--tserver_master_addrs=127.25.211.190:42401
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab
--principal=kudu/127.25.211.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:41907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:57:18.562815 23080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:57:18.563131 23080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:57:18.563261 23080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:57:18.570948 23080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:57:18.571077 23080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:57:18.571231 23080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.130
I20260806 02:57:18.581440 23080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:41907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.130.keytab
--principal=kudu/127.25.211.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.211.130
--webserver_port=0
--tserver_master_addrs=127.25.211.190:42401
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.130
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:57:18.582904 23080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:57:18.584309 23080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:57:18.598187 23093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:18.598254 23094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:18.598282 23097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:57:18.598902 23080 server_base.cc:1034] running on GCE node
I20260806 02:57:18.599658 23080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:57:18.600916 23080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:57:18.602205 23080 hybrid_clock.cc:648] HybridClock initialized: now 1785985038602112 us; error 50 us; skew 500 ppm
I20260806 02:57:18.602572 23080 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:57:18 dist-test-slave-zj8q krb5kdc[22729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985038, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:57:18.605902 23080 init.cc:377] Logged in from keytab as kudu/127.25.211.130@KRBTEST.COM (short username kudu)
I20260806 02:57:18.607542 23080 webserver.cc:460] Webserver started at http://127.25.211.130:42383/ using document root <none> and password file <none>
I20260806 02:57:18.608263 23080 fs_manager.cc:359] Metadata directory not provided
I20260806 02:57:18.608387 23080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:57:18.608733 23080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:57:18.611865 23080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/ts-1/data/instance:
uuid: "44c0216aca784dd0a3784f9692d114ee"
format_stamp: "Formatted at 2026-08-06 02:57:18 on dist-test-slave-zj8q"
server_key: "25193e29744a7a6a85898d657c018cb9"
server_key_iv: "caef7c31c603257379e4a20b38c03299"
server_key_version: "encryptionkey@0"
I20260806 02:57:18.612655 23080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/ts-1/wal/instance:
uuid: "44c0216aca784dd0a3784f9692d114ee"
format_stamp: "Formatted at 2026-08-06 02:57:18 on dist-test-slave-zj8q"
server_key: "25193e29744a7a6a85898d657c018cb9"
server_key_iv: "caef7c31c603257379e4a20b38c03299"
server_key_version: "encryptionkey@0"
I20260806 02:57:18.618016 23080 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260806 02:57:18.621724 23106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:18.623137 23080 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:57:18.623322 23080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/ts-1/wal
uuid: "44c0216aca784dd0a3784f9692d114ee"
format_stamp: "Formatted at 2026-08-06 02:57:18 on dist-test-slave-zj8q"
server_key: "25193e29744a7a6a85898d657c018cb9"
server_key_iv: "caef7c31c603257379e4a20b38c03299"
server_key_version: "encryptionkey@0"
I20260806 02:57:18.623521 23080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:57:18.647382 23080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:57:18.651765 23080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:57:18.652072 23080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:57:18.653411 23080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:57:18.654976 23080 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:57:18.655072 23080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:18.655206 23080 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:57:18.655272 23080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:18.687115 23080 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.130:38237
I20260806 02:57:18.687141 23259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.130:38237 every 8 connection(s)
I20260806 02:57:18.688465 23080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
I20260806 02:57:18.698091 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 23080
I20260806 02:57:18.698304 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/ts-1/wal/instance
I20260806 02:57:18.698943 26446 external_mini_cluster.cc:1428] Setting key 0f3314035e605040afa3a74f562ba693
Aug 06 02:57:18 dist-test-slave-zj8q krb5kdc[22729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985038, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.130@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:57:18.709688 23264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:18.691045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:52369 (local address 127.25.211.190:42401)
0806 02:57:18.691470 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:18.691475 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:18.692379 (+ 904us) server_negotiation.cc:407] Connection header received
0806 02:57:18.694258 (+ 1879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:18.694264 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:18.694356 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:18.694455 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:18.696019 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:18.696755 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:18.698379 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:18.699027 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:18.699177 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:18.703558 (+ 4381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:18.703564 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:18.703567 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:57:18.703604 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:18.706254 (+ 2650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:18.707208 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:18.707213 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:18.707215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:18.707374 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:18.707863 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:18.707867 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:18.707869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:18.708161 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:18.708277 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:18.709197 (+ 920us) server_negotiation.cc:299] Negotiation successful
0806 02:57:18.709383 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":210,"threads_started":1}
I20260806 02:57:18.711097 23265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:18.691428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:42401 (local address 127.25.211.130:52369)
0806 02:57:18.692200 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0806 02:57:18.692244 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 02:57:18.693948 (+ 1704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:57:18.694716 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:57:18.694724 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:57:18.695263 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:57:18.695835 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:18.695854 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:18.696905 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:18.696909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:18.698194 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:18.698211 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:18.699262 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:18.699267 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:18.699479 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:18.700714 (+ 1235us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:57:18.700745 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:57:18.703269 (+ 2524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:57:18.706427 (+ 3158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:57:18.706438 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:57:18.706457 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:57:18.707057 (+ 600us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:57:18.707479 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:57:18.707483 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:57:18.707486 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:57:18.707741 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:57:18.708296 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:57:18.708304 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:57:18.708762 (+ 458us) client_negotiation.cc:769] Sending connection context
0806 02:57:18.709093 (+ 331us) client_negotiation.cc:240] Negotiation successful
0806 02:57:18.709660 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":286,"threads_started":1}
I20260806 02:57:18.712581 23260 heartbeater.cc:347] Connected to a master server at 127.25.211.190:42401
I20260806 02:57:18.712894 23260 heartbeater.cc:464] Registering TS with master...
I20260806 02:57:18.713650 23260 heartbeater.cc:511] Master 127.25.211.190:42401 requested a full tablet report, sending...
I20260806 02:57:18.715250 22795 ts_manager.cc:194] Registered new tserver with Master: 44c0216aca784dd0a3784f9692d114ee (127.25.211.130:38237)
I20260806 02:57:18.716086 22795 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.211.130@KRBTEST.COM'} at 127.25.211.130:52369
WARNING: no policy specified for kudu/127.25.211.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab.
Entry for principal kudu/127.25.211.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab.
I20260806 02:57:18.769901 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.131:0
--local_ip_for_outbound_sockets=127.25.211.131
--webserver_interface=127.25.211.131
--webserver_port=0
--tserver_master_addrs=127.25.211.190:42401
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab
--principal=kudu/127.25.211.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:41907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:57:19.113094 23270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:57:19.113422 23270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:57:19.113559 23270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:57:19.121275 23270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:57:19.121419 23270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:57:19.121578 23270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.131
I20260806 02:57:19.132237 23270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:41907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.131.keytab
--principal=kudu/127.25.211.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.211.131
--webserver_port=0
--tserver_master_addrs=127.25.211.190:42401
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.131
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:57:19.133769 23270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:57:19.135195 23270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:57:19.148711 23278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:19.148689 23285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:19.148756 23279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:57:19.149873 23270 server_base.cc:1034] running on GCE node
I20260806 02:57:19.150592 23270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:57:19.151598 23270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:57:19.152834 23270 hybrid_clock.cc:648] HybridClock initialized: now 1785985039152764 us; error 61 us; skew 500 ppm
I20260806 02:57:19.153291 23270 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:57:19 dist-test-slave-zj8q krb5kdc[22729](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985039, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:57:19.156771 23270 init.cc:377] Logged in from keytab as kudu/127.25.211.131@KRBTEST.COM (short username kudu)
I20260806 02:57:19.158427 23270 webserver.cc:460] Webserver started at http://127.25.211.131:36743/ using document root <none> and password file <none>
I20260806 02:57:19.159173 23270 fs_manager.cc:359] Metadata directory not provided
I20260806 02:57:19.159299 23270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:57:19.159619 23270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:57:19.162792 23270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/ts-2/data/instance:
uuid: "804d49993dbc4e78b6c34ac8d15a2cba"
format_stamp: "Formatted at 2026-08-06 02:57:19 on dist-test-slave-zj8q"
server_key: "4ca0d4815803db84f0de89b165c47a37"
server_key_iv: "b463d4a8bcd1bfb80cab1f6ba4133321"
server_key_version: "encryptionkey@0"
I20260806 02:57:19.163632 23270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/ts-2/wal/instance:
uuid: "804d49993dbc4e78b6c34ac8d15a2cba"
format_stamp: "Formatted at 2026-08-06 02:57:19 on dist-test-slave-zj8q"
server_key: "4ca0d4815803db84f0de89b165c47a37"
server_key_iv: "b463d4a8bcd1bfb80cab1f6ba4133321"
server_key_version: "encryptionkey@0"
I20260806 02:57:19.169206 23270 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260806 02:57:19.172346 23073 heartbeater.cc:503] Master 127.25.211.190:42401 was elected leader, sending a full tablet report...
I20260806 02:57:19.173213 23293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:19.174963 23270 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 02:57:19.175156 23270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/ts-2/wal
uuid: "804d49993dbc4e78b6c34ac8d15a2cba"
format_stamp: "Formatted at 2026-08-06 02:57:19 on dist-test-slave-zj8q"
server_key: "4ca0d4815803db84f0de89b165c47a37"
server_key_iv: "b463d4a8bcd1bfb80cab1f6ba4133321"
server_key_version: "encryptionkey@0"
I20260806 02:57:19.175381 23270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:57:19.204487 23270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:57:19.208635 23270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:57:19.208907 23270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:57:19.209947 23270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:57:19.211352 23270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:57:19.211426 23270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:19.211540 23270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:57:19.211597 23270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:19.243021 23270 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.131:34789
I20260806 02:57:19.243073 23460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.131:34789 every 8 connection(s)
I20260806 02:57:19.244318 23270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
I20260806 02:57:19.254249 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 23270
I20260806 02:57:19.254428 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/ts-2/wal/instance
I20260806 02:57:19.254891 26446 external_mini_cluster.cc:1428] Setting key 668afeab7229f1aedaf4a39b4fee501d
Aug 06 02:57:19 dist-test-slave-zj8q krb5kdc[22729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985039, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.131@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:57:19.265003 23465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:19.247018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:34415 (local address 127.25.211.190:42401)
0806 02:57:19.247488 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:19.247493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:19.248321 (+ 828us) server_negotiation.cc:407] Connection header received
0806 02:57:19.250305 (+ 1984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:19.250311 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:19.250405 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:19.250507 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:19.252196 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:19.253017 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:19.254934 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:19.255400 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:19.255523 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:19.259478 (+ 3955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:19.259483 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:19.259487 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:57:19.259525 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:19.261936 (+ 2411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:19.262699 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:19.262703 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:19.262705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:19.262828 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:19.263252 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:19.263256 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:19.263258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:19.263537 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:19.263682 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:19.264399 (+ 717us) server_negotiation.cc:299] Negotiation successful
0806 02:57:19.264629 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":237,"threads_started":1}
I20260806 02:57:19.265965 23466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:19.247433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:42401 (local address 127.25.211.131:34415)
0806 02:57:19.248146 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0806 02:57:19.248186 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 02:57:19.249962 (+ 1776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:57:19.250798 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:57:19.250806 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:57:19.251383 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:57:19.251989 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:19.252008 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:19.253276 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:19.253290 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:19.254759 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:19.254779 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:19.255655 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:19.255663 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:19.255966 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:19.256657 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:57:19.256681 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:57:19.259257 (+ 2576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:57:19.262075 (+ 2818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:57:19.262085 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:57:19.262103 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:57:19.262573 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:57:19.262932 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:57:19.262937 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:57:19.262941 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:57:19.263132 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:57:19.263663 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:57:19.263671 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:57:19.264025 (+ 354us) client_negotiation.cc:769] Sending connection context
0806 02:57:19.264267 (+ 242us) client_negotiation.cc:240] Negotiation successful
0806 02:57:19.264825 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":275,"threads_started":1}
I20260806 02:57:19.267462 23461 heartbeater.cc:347] Connected to a master server at 127.25.211.190:42401
I20260806 02:57:19.267753 23461 heartbeater.cc:464] Registering TS with master...
I20260806 02:57:19.268462 23461 heartbeater.cc:511] Master 127.25.211.190:42401 requested a full tablet report, sending...
I20260806 02:57:19.269949 22795 ts_manager.cc:194] Registered new tserver with Master: 804d49993dbc4e78b6c34ac8d15a2cba (127.25.211.131:34789)
I20260806 02:57:19.270809 22795 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.25.211.131@KRBTEST.COM'} at 127.25.211.131:34415
I20260806 02:57:19.282750 26446 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:57:19.299088 23465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:19.287154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56958 (local address 127.25.211.190:42401)
0806 02:57:19.287357 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:19.287365 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:19.287779 (+ 414us) server_negotiation.cc:407] Connection header received
0806 02:57:19.287921 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:19.287925 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:19.287992 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:19.288074 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:19.289494 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:19.290267 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:19.291577 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:19.291989 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:19.292080 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:19.294263 (+ 2183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:19.294268 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:19.294271 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:57:19.294306 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:19.296672 (+ 2366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:19.297321 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:19.297324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:19.297326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:19.297436 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:19.297714 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:19.297717 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:19.297719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:19.298038 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:19.298250 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:19.298603 (+ 353us) server_negotiation.cc:299] Negotiation successful
0806 02:57:19.298798 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 02:57:19.302933 22795 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56958:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:57:19.309706 22795 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:57:19.331322 23481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:19.321291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:44779 (local address 127.0.0.1:45236)
0806 02:57:19.322408 (+ 1117us) negotiation.cc:107] Waiting for socket to connect
0806 02:57:19.322449 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 02:57:19.322611 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:57:19.323922 (+ 1311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:57:19.323929 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:57:19.323958 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:57:19.324342 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:19.324357 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:19.326587 (+ 2230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:19.326593 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:19.328743 (+ 2150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:19.328756 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:19.330329 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:19.330336 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:19.330539 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:19.330607 (+ 68us) client_negotiation.cc:769] Sending connection context
0806 02:57:19.330721 (+ 114us) client_negotiation.cc:240] Negotiation successful
0806 02:57:19.330921 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":842,"thread_start_us":338,"threads_started":1}
I20260806 02:57:19.332146 23479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:19.321543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.131:34789 (local address 127.0.0.1:60042)
0806 02:57:19.322992 (+ 1449us) negotiation.cc:107] Waiting for socket to connect
0806 02:57:19.323009 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 02:57:19.323110 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:57:19.325053 (+ 1943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:57:19.325061 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:57:19.325084 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:57:19.325458 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:19.325468 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:19.327142 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:19.327148 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:19.329489 (+ 2341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:19.329505 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:19.331272 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:19.331279 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:19.331458 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:19.331489 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 02:57:19.331577 (+ 88us) client_negotiation.cc:240] Negotiation successful
0806 02:57:19.331715 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1332,"mutex_wait_us":65,"thread_start_us":1000,"threads_started":1}
I20260806 02:57:19.332558 23480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:19.321618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45236 (local address 127.25.211.129:44779)
0806 02:57:19.323310 (+ 1692us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:19.323320 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:19.323349 (+ 29us) server_negotiation.cc:407] Connection header received
0806 02:57:19.323447 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:19.323454 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:19.323695 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:19.324535 (+ 840us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:57:19.324790 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:19.326388 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:19.328896 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:19.330063 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:19.331828 (+ 1765us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:19.331982 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:19.332060 (+ 78us) server_negotiation.cc:299] Negotiation successful
0806 02:57:19.332247 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1501,"thread_start_us":1575,"threads_started":1}
I20260806 02:57:19.332692 23485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:19.322606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.130:38237 (local address 127.0.0.1:53990)
0806 02:57:19.323579 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0806 02:57:19.323600 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 02:57:19.323700 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:57:19.324741 (+ 1041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:57:19.324746 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:57:19.324769 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:57:19.325198 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:19.325209 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:19.327502 (+ 2293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:19.327507 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:19.329675 (+ 2168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:19.329686 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:19.331659 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:19.331665 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:19.331889 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:19.331922 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 02:57:19.332020 (+ 98us) client_negotiation.cc:240] Negotiation successful
0806 02:57:19.332164 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":863,"mutex_wait_us":77,"thread_start_us":718,"threads_started":1}
I20260806 02:57:19.332867 23482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:19.322047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60042 (local address 127.25.211.131:34789)
0806 02:57:19.324499 (+ 2452us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:19.324506 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:19.324535 (+ 29us) server_negotiation.cc:407] Connection header received
0806 02:57:19.324625 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:19.324632 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:19.324847 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:19.325045 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:57:19.325622 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:19.326974 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:19.329670 (+ 2696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:19.331040 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:19.331243 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:19.331419 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:19.332397 (+ 978us) server_negotiation.cc:299] Negotiation successful
0806 02:57:19.332574 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2287,"thread_start_us":2170,"threads_started":1}
I20260806 02:57:19.332906 23483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:19.322655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53990 (local address 127.25.211.130:38237)
0806 02:57:19.324243 (+ 1588us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:19.324252 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:19.324279 (+ 27us) server_negotiation.cc:407] Connection header received
0806 02:57:19.324367 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:19.324374 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:19.324564 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:19.324728 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:57:19.325409 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:19.327329 (+ 1920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:19.329815 (+ 2486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:19.331190 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:19.331355 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:19.331515 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:19.332067 (+ 552us) server_negotiation.cc:299] Negotiation successful
0806 02:57:19.332431 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1446,"thread_start_us":1299,"threads_started":1}
I20260806 02:57:19.336520 23372 tablet_service.cc:1467] Processing CreateTablet for tablet edbfb8a74a80419b897a828f7607d172 (DEFAULT_TABLE table=test-table [id=576736c1fb224084b08d503234412714]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:57:19.336632 22982 tablet_service.cc:1467] Processing CreateTablet for tablet edbfb8a74a80419b897a828f7607d172 (DEFAULT_TABLE table=test-table [id=576736c1fb224084b08d503234412714]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:57:19.336622 23169 tablet_service.cc:1467] Processing CreateTablet for tablet edbfb8a74a80419b897a828f7607d172 (DEFAULT_TABLE table=test-table [id=576736c1fb224084b08d503234412714]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:57:19.337910 23372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edbfb8a74a80419b897a828f7607d172. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:57:19.337978 22982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edbfb8a74a80419b897a828f7607d172. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:57:19.337980 23169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edbfb8a74a80419b897a828f7607d172. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:57:19.350198 23486 tablet_bootstrap.cc:492] T edbfb8a74a80419b897a828f7607d172 P 804d49993dbc4e78b6c34ac8d15a2cba: Bootstrap starting.
I20260806 02:57:19.352849 23489 tablet_bootstrap.cc:492] T edbfb8a74a80419b897a828f7607d172 P 44c0216aca784dd0a3784f9692d114ee: Bootstrap starting.
I20260806 02:57:19.353668 23488 tablet_bootstrap.cc:492] T edbfb8a74a80419b897a828f7607d172 P 50eed71be1674a23b534853396f228b1: Bootstrap starting.
I20260806 02:57:19.355022 23486 tablet_bootstrap.cc:654] T edbfb8a74a80419b897a828f7607d172 P 804d49993dbc4e78b6c34ac8d15a2cba: Neither blocks nor log segments found. Creating new log.
I20260806 02:57:19.356050 23486 log.cc:824] T edbfb8a74a80419b897a828f7607d172 P 804d49993dbc4e78b6c34ac8d15a2cba: Log is configured to *not* fsync() on all Append() calls
I20260806 02:57:19.357349 23489 tablet_bootstrap.cc:654] T edbfb8a74a80419b897a828f7607d172 P 44c0216aca784dd0a3784f9692d114ee: Neither blocks nor log segments found. Creating new log.
I20260806 02:57:19.357348 23488 tablet_bootstrap.cc:654] T edbfb8a74a80419b897a828f7607d172 P 50eed71be1674a23b534853396f228b1: Neither blocks nor log segments found. Creating new log.
I20260806 02:57:19.358640 23489 log.cc:824] T edbfb8a74a80419b897a828f7607d172 P 44c0216aca784dd0a3784f9692d114ee: Log is configured to *not* fsync() on all Append() calls
I20260806 02:57:19.358641 23488 log.cc:824] T edbfb8a74a80419b897a828f7607d172 P 50eed71be1674a23b534853396f228b1: Log is configured to *not* fsync() on all Append() calls
I20260806 02:57:19.358850 23486 tablet_bootstrap.cc:492] T edbfb8a74a80419b897a828f7607d172 P 804d49993dbc4e78b6c34ac8d15a2cba: No bootstrap required, opened a new log
I20260806 02:57:19.359115 23486 ts_tablet_manager.cc:1397] T edbfb8a74a80419b897a828f7607d172 P 804d49993dbc4e78b6c34ac8d15a2cba: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260806 02:57:19.361707 23488 tablet_bootstrap.cc:492] T edbfb8a74a80419b897a828f7607d172 P 50eed71be1674a23b534853396f228b1: No bootstrap required, opened a new log
I20260806 02:57:19.362246 23488 ts_tablet_manager.cc:1397] T edbfb8a74a80419b897a828f7607d172 P 50eed71be1674a23b534853396f228b1: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.002s
I20260806 02:57:19.362682 23489 tablet_bootstrap.cc:492] T edbfb8a74a80419b897a828f7607d172 P 44c0216aca784dd0a3784f9692d114ee: No bootstrap required, opened a new log
I20260806 02:57:19.362998 23489 ts_tablet_manager.cc:1397] T edbfb8a74a80419b897a828f7607d172 P 44c0216aca784dd0a3784f9692d114ee: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260806 02:57:19.368112 23486 raft_consensus.cc:348] T edbfb8a74a80419b897a828f7607d172 P 804d49993dbc4e78b6c34ac8d15a2cba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50eed71be1674a23b534853396f228b1" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 44779 } } peers { permanent_uuid: "804d49993dbc4e78b6c34ac8d15a2cba" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 34789 } } peers { permanent_uuid: "44c0216aca784dd0a3784f9692d114ee" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 38237 } }
I20260806 02:57:19.368497 23486 raft_consensus.cc:374] T edbfb8a74a80419b897a828f7607d172 P 804d49993dbc4e78b6c34ac8d15a2cba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:57:19.368614 23486 raft_consensus.cc:729] T edbfb8a74a80419b897a828f7607d172 P 804d49993dbc4e78b6c34ac8d15a2cba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 804d49993dbc4e78b6c34ac8d15a2cba, State: Initialized, Role: FOLLOWER
I20260806 02:57:19.369225 23486 consensus_queue.cc:260] T edbfb8a74a80419b897a828f7607d172 P 804d49993dbc4e78b6c34ac8d15a2cba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50eed71be1674a23b534853396f228b1" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 44779 } } peers { permanent_uuid: "804d49993dbc4e78b6c34ac8d15a2cba" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 34789 } } peers { permanent_uuid: "44c0216aca784dd0a3784f9692d114ee" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 38237 } }
I20260806 02:57:19.370397 23461 heartbeater.cc:503] Master 127.25.211.190:42401 was elected leader, sending a full tablet report...
I20260806 02:57:19.370889 23486 ts_tablet_manager.cc:1428] T edbfb8a74a80419b897a828f7607d172 P 804d49993dbc4e78b6c34ac8d15a2cba: Time spent starting tablet: real 0.012s user 0.010s sys 0.001s
I20260806 02:57:19.373001 23489 raft_consensus.cc:348] T edbfb8a74a80419b897a828f7607d172 P 44c0216aca784dd0a3784f9692d114ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50eed71be1674a23b534853396f228b1" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 44779 } } peers { permanent_uuid: "804d49993dbc4e78b6c34ac8d15a2cba" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 34789 } } peers { permanent_uuid: "44c0216aca784dd0a3784f9692d114ee" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 38237 } }
I20260806 02:57:19.373560 23489 raft_consensus.cc:374] T edbfb8a74a80419b897a828f7607d172 P 44c0216aca784dd0a3784f9692d114ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:57:19.373319 23488 raft_consensus.cc:348] T edbfb8a74a80419b897a828f7607d172 P 50eed71be1674a23b534853396f228b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50eed71be1674a23b534853396f228b1" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 44779 } } peers { permanent_uuid: "804d49993dbc4e78b6c34ac8d15a2cba" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 34789 } } peers { permanent_uuid: "44c0216aca784dd0a3784f9692d114ee" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 38237 } }
I20260806 02:57:19.373667 23489 raft_consensus.cc:729] T edbfb8a74a80419b897a828f7607d172 P 44c0216aca784dd0a3784f9692d114ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44c0216aca784dd0a3784f9692d114ee, State: Initialized, Role: FOLLOWER
I20260806 02:57:19.373800 23488 raft_consensus.cc:374] T edbfb8a74a80419b897a828f7607d172 P 50eed71be1674a23b534853396f228b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:57:19.373932 23488 raft_consensus.cc:729] T edbfb8a74a80419b897a828f7607d172 P 50eed71be1674a23b534853396f228b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50eed71be1674a23b534853396f228b1, State: Initialized, Role: FOLLOWER
I20260806 02:57:19.374471 23489 consensus_queue.cc:260] T edbfb8a74a80419b897a828f7607d172 P 44c0216aca784dd0a3784f9692d114ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50eed71be1674a23b534853396f228b1" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 44779 } } peers { permanent_uuid: "804d49993dbc4e78b6c34ac8d15a2cba" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 34789 } } peers { permanent_uuid: "44c0216aca784dd0a3784f9692d114ee" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 38237 } }
I20260806 02:57:19.374550 23488 consensus_queue.cc:260] T edbfb8a74a80419b897a828f7607d172 P 50eed71be1674a23b534853396f228b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50eed71be1674a23b534853396f228b1" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 44779 } } peers { permanent_uuid: "804d49993dbc4e78b6c34ac8d15a2cba" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 34789 } } peers { permanent_uuid: "44c0216aca784dd0a3784f9692d114ee" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 38237 } }
I20260806 02:57:19.376387 23488 ts_tablet_manager.cc:1428] T edbfb8a74a80419b897a828f7607d172 P 50eed71be1674a23b534853396f228b1: Time spent starting tablet: real 0.014s user 0.010s sys 0.002s
I20260806 02:57:19.377488 23260 heartbeater.cc:503] Master 127.25.211.190:42401 was elected leader, sending a full tablet report...
I20260806 02:57:19.378031 23489 ts_tablet_manager.cc:1428] T edbfb8a74a80419b897a828f7607d172 P 44c0216aca784dd0a3784f9692d114ee: Time spent starting tablet: real 0.015s user 0.006s sys 0.008s
I20260806 02:57:19.384572 23494 raft_consensus.cc:482] T edbfb8a74a80419b897a828f7607d172 P 44c0216aca784dd0a3784f9692d114ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:57:19.384830 23494 raft_consensus.cc:504] T edbfb8a74a80419b897a828f7607d172 P 44c0216aca784dd0a3784f9692d114ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50eed71be1674a23b534853396f228b1" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 44779 } } peers { permanent_uuid: "804d49993dbc4e78b6c34ac8d15a2cba" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 34789 } } peers { permanent_uuid: "44c0216aca784dd0a3784f9692d114ee" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 38237 } }
I20260806 02:57:19.386618 23494 leader_election.cc:288] T edbfb8a74a80419b897a828f7607d172 P 44c0216aca784dd0a3784f9692d114ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50eed71be1674a23b534853396f228b1 (127.25.211.129:44779), 804d49993dbc4e78b6c34ac8d15a2cba (127.25.211.131:34789)
W20260806 02:57:19.387723 23074 tablet.cc:2369] T edbfb8a74a80419b897a828f7607d172 P 50eed71be1674a23b534853396f228b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:57:19.395740 23480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:19.387984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:36205 (local address 127.25.211.129:44779)
0806 02:57:19.388195 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:19.388201 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:19.388221 (+ 20us) server_negotiation.cc:407] Connection header received
0806 02:57:19.388289 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:19.388294 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:19.388362 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:19.388459 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:57:19.389155 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:19.390323 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:19.393257 (+ 2934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:19.394587 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:19.394728 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:19.394803 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:19.395321 (+ 518us) server_negotiation.cc:299] Negotiation successful
0806 02:57:19.395463 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 02:57:19.395720 23482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:19.387456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:48529 (local address 127.25.211.131:34789)
0806 02:57:19.387647 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:19.387653 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:19.387898 (+ 245us) server_negotiation.cc:407] Connection header received
0806 02:57:19.388170 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:19.388176 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:19.388257 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:19.388321 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:57:19.389193 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:19.390323 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:19.393262 (+ 2939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:19.394548 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:19.394728 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:19.394803 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:19.395321 (+ 518us) server_negotiation.cc:299] Negotiation successful
0806 02:57:19.395463 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 02:57:19.396087 23496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:19.386791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:44779 (local address 127.25.211.130:36205)
0806 02:57:19.387518 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0806 02:57:19.387558 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 02:57:19.387717 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:57:19.388492 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:57:19.388496 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:57:19.388523 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:57:19.389008 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:19.389021 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:19.390439 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:19.390444 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:19.393061 (+ 2617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:19.393079 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:19.394827 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:19.394832 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:19.395068 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:19.395168 (+ 100us) client_negotiation.cc:769] Sending connection context
0806 02:57:19.395270 (+ 102us) client_negotiation.cc:240] Negotiation successful
0806 02:57:19.395438 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"mutex_wait_us":216,"thread_start_us":370,"threads_started":1}
I20260806 02:57:19.396138 23499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:19.386948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.131:34789 (local address 127.25.211.130:48529)
0806 02:57:19.387802 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0806 02:57:19.387818 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:57:19.387997 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:57:19.388512 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:57:19.388516 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:57:19.388533 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:57:19.389020 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:19.389030 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:19.390442 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:19.390446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:19.393061 (+ 2615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:19.393079 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:19.394822 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:19.394830 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:19.395067 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:19.395169 (+ 102us) client_negotiation.cc:769] Sending connection context
0806 02:57:19.395270 (+ 101us) client_negotiation.cc:240] Negotiation successful
0806 02:57:19.395438 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":729,"mutex_wait_us":184,"thread_start_us":1089,"threads_started":1}
I20260806 02:57:19.396962 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edbfb8a74a80419b897a828f7607d172" candidate_uuid: "44c0216aca784dd0a3784f9692d114ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "804d49993dbc4e78b6c34ac8d15a2cba" is_pre_election: true
I20260806 02:57:19.397055 23011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edbfb8a74a80419b897a828f7607d172" candidate_uuid: "44c0216aca784dd0a3784f9692d114ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50eed71be1674a23b534853396f228b1" is_pre_election: true
I20260806 02:57:19.397473 23395 raft_consensus.cc:2454] T edbfb8a74a80419b897a828f7607d172 P 804d49993dbc4e78b6c34ac8d15a2cba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44c0216aca784dd0a3784f9692d114ee in term 0.
I20260806 02:57:19.397518 23011 raft_consensus.cc:2454] T edbfb8a74a80419b897a828f7607d172 P 50eed71be1674a23b534853396f228b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44c0216aca784dd0a3784f9692d114ee in term 0.
I20260806 02:57:19.398131 23113 leader_election.cc:302] T edbfb8a74a80419b897a828f7607d172 P 44c0216aca784dd0a3784f9692d114ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 44c0216aca784dd0a3784f9692d114ee, 50eed71be1674a23b534853396f228b1, 804d49993dbc4e78b6c34ac8d15a2cba; no voters:
I20260806 02:57:19.398603 23494 raft_consensus.cc:2781] T edbfb8a74a80419b897a828f7607d172 P 44c0216aca784dd0a3784f9692d114ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:57:19.398729 23494 raft_consensus.cc:482] T edbfb8a74a80419b897a828f7607d172 P 44c0216aca784dd0a3784f9692d114ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:57:19.398833 23494 raft_consensus.cc:3037] T edbfb8a74a80419b897a828f7607d172 P 44c0216aca784dd0a3784f9692d114ee [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:57:19.401080 23494 raft_consensus.cc:504] T edbfb8a74a80419b897a828f7607d172 P 44c0216aca784dd0a3784f9692d114ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50eed71be1674a23b534853396f228b1" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 44779 } } peers { permanent_uuid: "804d49993dbc4e78b6c34ac8d15a2cba" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 34789 } } peers { permanent_uuid: "44c0216aca784dd0a3784f9692d114ee" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 38237 } }
I20260806 02:57:19.401705 23494 leader_election.cc:288] T edbfb8a74a80419b897a828f7607d172 P 44c0216aca784dd0a3784f9692d114ee [CANDIDATE]: Term 1 election: Requested vote from peers 50eed71be1674a23b534853396f228b1 (127.25.211.129:44779), 804d49993dbc4e78b6c34ac8d15a2cba (127.25.211.131:34789)
I20260806 02:57:19.402326 23011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edbfb8a74a80419b897a828f7607d172" candidate_uuid: "44c0216aca784dd0a3784f9692d114ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50eed71be1674a23b534853396f228b1"
I20260806 02:57:19.402357 23395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edbfb8a74a80419b897a828f7607d172" candidate_uuid: "44c0216aca784dd0a3784f9692d114ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "804d49993dbc4e78b6c34ac8d15a2cba"
I20260806 02:57:19.402540 23395 raft_consensus.cc:3037] T edbfb8a74a80419b897a828f7607d172 P 804d49993dbc4e78b6c34ac8d15a2cba [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:57:19.402572 23011 raft_consensus.cc:3037] T edbfb8a74a80419b897a828f7607d172 P 50eed71be1674a23b534853396f228b1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:57:19.405092 23395 raft_consensus.cc:2454] T edbfb8a74a80419b897a828f7607d172 P 804d49993dbc4e78b6c34ac8d15a2cba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44c0216aca784dd0a3784f9692d114ee in term 1.
I20260806 02:57:19.405113 23011 raft_consensus.cc:2454] T edbfb8a74a80419b897a828f7607d172 P 50eed71be1674a23b534853396f228b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44c0216aca784dd0a3784f9692d114ee in term 1.
I20260806 02:57:19.405499 23113 leader_election.cc:302] T edbfb8a74a80419b897a828f7607d172 P 44c0216aca784dd0a3784f9692d114ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44c0216aca784dd0a3784f9692d114ee, 804d49993dbc4e78b6c34ac8d15a2cba; no voters:
I20260806 02:57:19.405838 23494 raft_consensus.cc:2781] T edbfb8a74a80419b897a828f7607d172 P 44c0216aca784dd0a3784f9692d114ee [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:57:19.406354 23494 raft_consensus.cc:686] T edbfb8a74a80419b897a828f7607d172 P 44c0216aca784dd0a3784f9692d114ee [term 1 LEADER]: Becoming Leader. State: Replica: 44c0216aca784dd0a3784f9692d114ee, State: Running, Role: LEADER
I20260806 02:57:19.406847 23494 consensus_queue.cc:237] T edbfb8a74a80419b897a828f7607d172 P 44c0216aca784dd0a3784f9692d114ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50eed71be1674a23b534853396f228b1" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 44779 } } peers { permanent_uuid: "804d49993dbc4e78b6c34ac8d15a2cba" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 34789 } } peers { permanent_uuid: "44c0216aca784dd0a3784f9692d114ee" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 38237 } }
I20260806 02:57:19.411645 22795 catalog_manager.cc:5495] T edbfb8a74a80419b897a828f7607d172 P 44c0216aca784dd0a3784f9692d114ee reported cstate change: term changed from 0 to 1, leader changed from <none> to 44c0216aca784dd0a3784f9692d114ee (127.25.211.130). New cstate: current_term: 1 leader_uuid: "44c0216aca784dd0a3784f9692d114ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50eed71be1674a23b534853396f228b1" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 44779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "804d49993dbc4e78b6c34ac8d15a2cba" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 34789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44c0216aca784dd0a3784f9692d114ee" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 38237 } health_report { overall_health: HEALTHY } } }
I20260806 02:57:19.435299 23483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:19.429927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53998 (local address 127.25.211.130:38237)
0806 02:57:19.430187 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:19.430193 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:19.430300 (+ 107us) server_negotiation.cc:407] Connection header received
0806 02:57:19.430445 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:19.430449 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:19.430584 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:19.430724 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:57:19.431049 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:19.431817 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:19.433164 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:19.433604 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:19.433700 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:19.434221 (+ 521us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:57:19.434757 (+ 536us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:57:19.434873 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:19.434951 (+ 78us) server_negotiation.cc:299] Negotiation successful
0806 02:57:19.435017 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260806 02:57:19.440493 23262 tablet.cc:2369] T edbfb8a74a80419b897a828f7607d172 P 44c0216aca784dd0a3784f9692d114ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:57:19.444231 23395 raft_consensus.cc:1261] T edbfb8a74a80419b897a828f7607d172 P 804d49993dbc4e78b6c34ac8d15a2cba [term 1 FOLLOWER]: Refusing update from remote peer 44c0216aca784dd0a3784f9692d114ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:57:19.444235 23011 raft_consensus.cc:1261] T edbfb8a74a80419b897a828f7607d172 P 50eed71be1674a23b534853396f228b1 [term 1 FOLLOWER]: Refusing update from remote peer 44c0216aca784dd0a3784f9692d114ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:57:19.445245 23500 consensus_queue.cc:1035] T edbfb8a74a80419b897a828f7607d172 P 44c0216aca784dd0a3784f9692d114ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "50eed71be1674a23b534853396f228b1" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 44779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:57:19.445647 23494 consensus_queue.cc:1035] T edbfb8a74a80419b897a828f7607d172 P 44c0216aca784dd0a3784f9692d114ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "804d49993dbc4e78b6c34ac8d15a2cba" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 34789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:57:19.458696 23508 mvcc.cc:205] Tried to move back new op lower bound from 7315394721552928768 to 7315394721414561792. Current Snapshot: MvccSnapshot[applied={T|T < 7315394721552928768}]
I20260806 02:57:19.470784 22794 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:57:19.471129 22794 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:57:19.475037 22794 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37eb4d9ea59d4ebdb00aeffbdd541dcb: Sending DeleteTablet for 3 replicas of tablet edbfb8a74a80419b897a828f7607d172
I20260806 02:57:19.476615 23169 tablet_service.cc:1514] Processing DeleteTablet for tablet edbfb8a74a80419b897a828f7607d172 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:57:19 UTC) from {username='kudu', principal='kudu/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:53990
I20260806 02:57:19.477664 23372 tablet_service.cc:1514] Processing DeleteTablet for tablet edbfb8a74a80419b897a828f7607d172 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:57:19 UTC) from {username='kudu', principal='kudu/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:60042
I20260806 02:57:19.477766 22982 tablet_service.cc:1514] Processing DeleteTablet for tablet edbfb8a74a80419b897a828f7607d172 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:57:19 UTC) from {username='kudu', principal='kudu/127.25.211.190@KRBTEST.COM'} at 127.0.0.1:45236
I20260806 02:57:19.478909 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 22883
I20260806 02:57:19.488821 23522 tablet_replica.cc:330] T edbfb8a74a80419b897a828f7607d172 P 50eed71be1674a23b534853396f228b1: stopping tablet replica
I20260806 02:57:19.489523 23522 raft_consensus.cc:2229] T edbfb8a74a80419b897a828f7607d172 P 50eed71be1674a23b534853396f228b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:57:19.490267 23522 raft_consensus.cc:2258] T edbfb8a74a80419b897a828f7607d172 P 50eed71be1674a23b534853396f228b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:57:19.490931 23519 tablet_replica.cc:330] T edbfb8a74a80419b897a828f7607d172 P 44c0216aca784dd0a3784f9692d114ee: stopping tablet replica
I20260806 02:57:19.491791 23519 raft_consensus.cc:2229] T edbfb8a74a80419b897a828f7607d172 P 44c0216aca784dd0a3784f9692d114ee [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:57:19.494267 23522 ts_tablet_manager.cc:1905] T edbfb8a74a80419b897a828f7607d172 P 50eed71be1674a23b534853396f228b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:57:19.494668 23521 tablet_replica.cc:330] T edbfb8a74a80419b897a828f7607d172 P 804d49993dbc4e78b6c34ac8d15a2cba: stopping tablet replica
I20260806 02:57:19.495038 23519 raft_consensus.cc:2258] T edbfb8a74a80419b897a828f7607d172 P 44c0216aca784dd0a3784f9692d114ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:57:19.495039 23521 raft_consensus.cc:2229] T edbfb8a74a80419b897a828f7607d172 P 804d49993dbc4e78b6c34ac8d15a2cba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:57:19.495527 23521 raft_consensus.cc:2258] T edbfb8a74a80419b897a828f7607d172 P 804d49993dbc4e78b6c34ac8d15a2cba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:57:19.497376 23521 ts_tablet_manager.cc:1905] T edbfb8a74a80419b897a828f7607d172 P 804d49993dbc4e78b6c34ac8d15a2cba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:57:19.502012 23521 ts_tablet_manager.cc:1918] T edbfb8a74a80419b897a828f7607d172 P 804d49993dbc4e78b6c34ac8d15a2cba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:57:19.502209 23521 log.cc:1196] T edbfb8a74a80419b897a828f7607d172 P 804d49993dbc4e78b6c34ac8d15a2cba: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/ts-2/wal/wals/edbfb8a74a80419b897a828f7607d172
I20260806 02:57:19.502509 23522 ts_tablet_manager.cc:1918] T edbfb8a74a80419b897a828f7607d172 P 50eed71be1674a23b534853396f228b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:57:19.502701 23522 log.cc:1196] T edbfb8a74a80419b897a828f7607d172 P 50eed71be1674a23b534853396f228b1: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/ts-0/wal/wals/edbfb8a74a80419b897a828f7607d172
I20260806 02:57:19.502975 23521 ts_tablet_manager.cc:1939] T edbfb8a74a80419b897a828f7607d172 P 804d49993dbc4e78b6c34ac8d15a2cba: Deleting consensus metadata
I20260806 02:57:19.503264 23522 ts_tablet_manager.cc:1939] T edbfb8a74a80419b897a828f7607d172 P 50eed71be1674a23b534853396f228b1: Deleting consensus metadata
I20260806 02:57:19.505059 22776 catalog_manager.cc:4841] TS 804d49993dbc4e78b6c34ac8d15a2cba (127.25.211.131:34789): tablet edbfb8a74a80419b897a828f7607d172 (table test-table [id=576736c1fb224084b08d503234412714]) successfully deleted
I20260806 02:57:19.505137 22771 catalog_manager.cc:4841] TS 50eed71be1674a23b534853396f228b1 (127.25.211.129:44779): tablet edbfb8a74a80419b897a828f7607d172 (table test-table [id=576736c1fb224084b08d503234412714]) successfully deleted
I20260806 02:57:19.507524 23519 ts_tablet_manager.cc:1905] T edbfb8a74a80419b897a828f7607d172 P 44c0216aca784dd0a3784f9692d114ee: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 06 02:57:19 dist-test-slave-zj8q krb5kdc[22729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.129@KRBTEST.COM
I20260806 02:57:19.515373 23519 ts_tablet_manager.cc:1918] T edbfb8a74a80419b897a828f7607d172 P 44c0216aca784dd0a3784f9692d114ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:57:19.515597 23519 log.cc:1196] T edbfb8a74a80419b897a828f7607d172 P 44c0216aca784dd0a3784f9692d114ee: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1785984707115702-26446-0/minicluster-data/ts-1/wal/wals/edbfb8a74a80419b897a828f7607d172
I20260806 02:57:19.516171 23519 ts_tablet_manager.cc:1939] T edbfb8a74a80419b897a828f7607d172 P 44c0216aca784dd0a3784f9692d114ee: Deleting consensus metadata
I20260806 02:57:19.517596 23480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:19.479636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45246 (local address 127.25.211.129:44779)
0806 02:57:19.479813 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:19.479820 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:19.491242 (+ 11422us) server_negotiation.cc:407] Connection header received
0806 02:57:19.491558 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:19.491563 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:19.491649 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:19.491747 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:19.493637 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:19.494751 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:19.506063 (+ 11312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:19.506785 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:19.506930 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:19.511711 (+ 4781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:19.511724 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:19.511739 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:57:19.511781 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:19.514591 (+ 2810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:19.515354 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:19.515361 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:19.515364 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:19.515517 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:19.515954 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:19.515960 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:19.515964 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:19.516382 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:19.516680 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:19.517010 (+ 330us) server_negotiation.cc:299] Negotiation successful
0806 02:57:19.517242 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 02:57:19.518229 22774 catalog_manager.cc:4841] TS 44c0216aca784dd0a3784f9692d114ee (127.25.211.130:38237): tablet edbfb8a74a80419b897a828f7607d172 (table test-table [id=576736c1fb224084b08d503234412714]) successfully deleted
I20260806 02:57:19.518347 23066 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:57:19.615303 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 22883
I20260806 02:57:19.639431 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 23080
Aug 06 02:57:19 dist-test-slave-zj8q krb5kdc[22729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.130@KRBTEST.COM
I20260806 02:57:19.657624 23483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:19.640229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54014 (local address 127.25.211.130:38237)
0806 02:57:19.640394 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:19.640398 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:19.640420 (+ 22us) server_negotiation.cc:407] Connection header received
0806 02:57:19.640592 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:19.640596 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:19.640669 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:19.640781 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:19.642575 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:19.643455 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:19.645365 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:19.645851 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:19.645953 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:19.651666 (+ 5713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:19.651685 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:19.651704 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:57:19.651754 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:19.654808 (+ 3054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:19.655654 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:19.655661 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:19.655664 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:19.655813 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:19.656203 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:19.656206 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:19.656208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:19.656577 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:19.656858 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:19.657161 (+ 303us) server_negotiation.cc:299] Negotiation successful
0806 02:57:19.657338 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 02:57:19.658443 23254 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:57:19.773386 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 23080
I20260806 02:57:19.798100 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 23270
Aug 06 02:57:19 dist-test-slave-zj8q krb5kdc[22729](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.131@KRBTEST.COM
I20260806 02:57:19.814040 23482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:19.798928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60050 (local address 127.25.211.131:34789)
0806 02:57:19.799140 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:19.799145 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:19.799166 (+ 21us) server_negotiation.cc:407] Connection header received
0806 02:57:19.799279 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:19.799283 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:19.799349 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:19.799432 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:19.800853 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:19.801615 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:19.803129 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:19.803556 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:19.803681 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:19.808364 (+ 4683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:19.808383 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:19.808402 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:57:19.808459 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:19.811348 (+ 2889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:19.812174 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:19.812182 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:19.812184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:19.812337 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:19.812650 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:19.812653 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:19.812655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:19.813041 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:19.813278 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:19.813589 (+ 311us) server_negotiation.cc:299] Negotiation successful
0806 02:57:19.813782 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260806 02:57:19.814808 23456 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:57:19.905229 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 23270
I20260806 02:57:19.926091 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 22750
I20260806 02:57:19.927320 22832 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:57:20.042367 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 22750
2026-08-06T02:57:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4611 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-06T02:57:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:57:20Z Disabled control of system clock
I20260806 02:57:20.141983 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:35391
--webserver_interface=127.25.211.190
--webserver_port=0
--builtin_ntp_servers=127.25.211.148:39137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.211.190:35391
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 02:57:20.496892 23534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:57:20.497237 23534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:57:20.497324 23534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:57:20.505002 23534 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:57:20.505138 23534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:57:20.505177 23534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:57:20.505244 23534 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:57:20.505287 23534 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:57:20.514810 23534 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:39137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.211.190:35391
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:35391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.211.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:57:20.516059 23534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:57:20.517832 23534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:57:20.529623 23542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:20.529605 23543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:20.529605 23546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:57:20.530241 23534 server_base.cc:1034] running on GCE node
I20260806 02:57:20.531152 23534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:57:20.532620 23534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:57:20.533846 23534 hybrid_clock.cc:648] HybridClock initialized: now 1785985040533796 us; error 43 us; skew 500 ppm
I20260806 02:57:20.534271 23534 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:57:20.536691 23534 webserver.cc:460] Webserver started at http://127.25.211.190:35199/ using document root <none> and password file <none>
I20260806 02:57:20.537384 23534 fs_manager.cc:359] Metadata directory not provided
I20260806 02:57:20.537489 23534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:57:20.537832 23534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:57:20.540830 23534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/master-0/data/instance:
uuid: "e92c1dc693d549c797c36c7f08faf276"
format_stamp: "Formatted at 2026-08-06 02:57:20 on dist-test-slave-zj8q"
server_key: "f9424cab4721321c03393cfe394c4c89"
server_key_iv: "377539754efb1646d9c4ec857da56cde"
server_key_version: "encryptionkey@0"
I20260806 02:57:20.541626 23534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/master-0/wal/instance:
uuid: "e92c1dc693d549c797c36c7f08faf276"
format_stamp: "Formatted at 2026-08-06 02:57:20 on dist-test-slave-zj8q"
server_key: "f9424cab4721321c03393cfe394c4c89"
server_key_iv: "377539754efb1646d9c4ec857da56cde"
server_key_version: "encryptionkey@0"
I20260806 02:57:20.547066 23534 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260806 02:57:20.550632 23555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:20.552001 23534 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260806 02:57:20.552181 23534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/master-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/master-0/wal
uuid: "e92c1dc693d549c797c36c7f08faf276"
format_stamp: "Formatted at 2026-08-06 02:57:20 on dist-test-slave-zj8q"
server_key: "f9424cab4721321c03393cfe394c4c89"
server_key_iv: "377539754efb1646d9c4ec857da56cde"
server_key_version: "encryptionkey@0"
I20260806 02:57:20.552392 23534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:57:20.577523 23534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:57:20.578521 23534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:57:20.578826 23534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:57:20.597450 23534 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.190:35391
I20260806 02:57:20.597476 23634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.190:35391 every 8 connection(s)
I20260806 02:57:20.598917 23534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
I20260806 02:57:20.602926 23635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:57:20.606402 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 23534
I20260806 02:57:20.606587 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/master-0/wal/instance
I20260806 02:57:20.607194 26446 external_mini_cluster.cc:1428] Setting key d36866816d0b1836291316d4136666a3
I20260806 02:57:20.616055 23635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e92c1dc693d549c797c36c7f08faf276: Bootstrap starting.
I20260806 02:57:20.619356 23635 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e92c1dc693d549c797c36c7f08faf276: Neither blocks nor log segments found. Creating new log.
I20260806 02:57:20.620528 23635 log.cc:824] T 00000000000000000000000000000000 P e92c1dc693d549c797c36c7f08faf276: Log is configured to *not* fsync() on all Append() calls
I20260806 02:57:20.621274 23639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:20.609332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38158 (local address 127.25.211.190:35391)
0806 02:57:20.610399 (+ 1067us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:20.610414 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:20.610463 (+ 49us) server_negotiation.cc:407] Connection header received
0806 02:57:20.612313 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:20.612344 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:20.612885 (+ 541us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:20.613331 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:20.613957 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:20.615351 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:20.616753 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:20.617461 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:20.617666 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:20.618771 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:20.618779 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:20.618795 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:57:20.619350 (+ 555us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:20.619459 (+ 109us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:57:20.619487 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:20.619594 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:20.619839 (+ 245us) server_negotiation.cc:299] Negotiation successful
0806 02:57:20.620153 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":350,"threads_started":1}
I20260806 02:57:20.623939 23635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e92c1dc693d549c797c36c7f08faf276: No bootstrap required, opened a new log
I20260806 02:57:20.632158 23635 raft_consensus.cc:348] T 00000000000000000000000000000000 P e92c1dc693d549c797c36c7f08faf276 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e92c1dc693d549c797c36c7f08faf276" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 35391 } }
I20260806 02:57:20.632573 23635 raft_consensus.cc:374] T 00000000000000000000000000000000 P e92c1dc693d549c797c36c7f08faf276 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:57:20.632650 23635 raft_consensus.cc:729] T 00000000000000000000000000000000 P e92c1dc693d549c797c36c7f08faf276 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e92c1dc693d549c797c36c7f08faf276, State: Initialized, Role: FOLLOWER
I20260806 02:57:20.633193 23635 consensus_queue.cc:260] T 00000000000000000000000000000000 P e92c1dc693d549c797c36c7f08faf276 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e92c1dc693d549c797c36c7f08faf276" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 35391 } }
I20260806 02:57:20.633419 23635 raft_consensus.cc:388] T 00000000000000000000000000000000 P e92c1dc693d549c797c36c7f08faf276 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:57:20.633525 23635 raft_consensus.cc:482] T 00000000000000000000000000000000 P e92c1dc693d549c797c36c7f08faf276 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:57:20.633657 23635 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e92c1dc693d549c797c36c7f08faf276 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:57:20.635994 23635 raft_consensus.cc:504] T 00000000000000000000000000000000 P e92c1dc693d549c797c36c7f08faf276 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e92c1dc693d549c797c36c7f08faf276" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 35391 } }
I20260806 02:57:20.636520 23635 leader_election.cc:302] T 00000000000000000000000000000000 P e92c1dc693d549c797c36c7f08faf276 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e92c1dc693d549c797c36c7f08faf276; no voters:
I20260806 02:57:20.637189 23635 leader_election.cc:288] T 00000000000000000000000000000000 P e92c1dc693d549c797c36c7f08faf276 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:57:20.637348 23642 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e92c1dc693d549c797c36c7f08faf276 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:57:20.638335 23642 raft_consensus.cc:686] T 00000000000000000000000000000000 P e92c1dc693d549c797c36c7f08faf276 [term 1 LEADER]: Becoming Leader. State: Replica: e92c1dc693d549c797c36c7f08faf276, State: Running, Role: LEADER
I20260806 02:57:20.638907 23642 consensus_queue.cc:237] T 00000000000000000000000000000000 P e92c1dc693d549c797c36c7f08faf276 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e92c1dc693d549c797c36c7f08faf276" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 35391 } }
I20260806 02:57:20.639592 23635 sys_catalog.cc:564] T 00000000000000000000000000000000 P e92c1dc693d549c797c36c7f08faf276 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:57:20.641716 23645 sys_catalog.cc:455] T 00000000000000000000000000000000 P e92c1dc693d549c797c36c7f08faf276 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e92c1dc693d549c797c36c7f08faf276. Latest consensus state: current_term: 1 leader_uuid: "e92c1dc693d549c797c36c7f08faf276" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e92c1dc693d549c797c36c7f08faf276" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 35391 } } }
I20260806 02:57:20.642225 23645 sys_catalog.cc:458] T 00000000000000000000000000000000 P e92c1dc693d549c797c36c7f08faf276 [sys.catalog]: This master's current role is: LEADER
I20260806 02:57:20.642069 23644 sys_catalog.cc:455] T 00000000000000000000000000000000 P e92c1dc693d549c797c36c7f08faf276 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e92c1dc693d549c797c36c7f08faf276" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e92c1dc693d549c797c36c7f08faf276" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 35391 } } }
I20260806 02:57:20.642503 23644 sys_catalog.cc:458] T 00000000000000000000000000000000 P e92c1dc693d549c797c36c7f08faf276 [sys.catalog]: This master's current role is: LEADER
I20260806 02:57:20.644188 23665 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:57:20.648803 23665 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:57:20.657624 23665 catalog_manager.cc:1347] Generated new cluster ID: cb2ee2436e1746958e90d1bb967ec472
I20260806 02:57:20.657747 23665 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:57:20.684850 23665 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:57:20.686828 23665 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:57:20.701218 23665 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e92c1dc693d549c797c36c7f08faf276: Generated new TSK 0
I20260806 02:57:20.702325 23665 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:57:20.721769 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.129:0
--local_ip_for_outbound_sockets=127.25.211.129
--webserver_interface=127.25.211.129
--webserver_port=0
--tserver_master_addrs=127.25.211.190:35391
--builtin_ntp_servers=127.25.211.148:39137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 02:57:21.083074 23679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:57:21.083385 23679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:57:21.083515 23679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:57:21.090991 23679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:57:21.091116 23679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:57:21.091269 23679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.129
I20260806 02:57:21.100340 23679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:39137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.211.129
--webserver_port=0
--tserver_master_addrs=127.25.211.190:35391
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.129
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:57:21.101665 23679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:57:21.103574 23679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:57:21.117954 23684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:21.117399 23685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:21.117391 23687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:57:21.119115 23679 server_base.cc:1034] running on GCE node
I20260806 02:57:21.119833 23679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:57:21.121083 23679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:57:21.122376 23679 hybrid_clock.cc:648] HybridClock initialized: now 1785985041122305 us; error 68 us; skew 500 ppm
I20260806 02:57:21.122803 23679 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:57:21.128463 23679 webserver.cc:460] Webserver started at http://127.25.211.129:46775/ using document root <none> and password file <none>
I20260806 02:57:21.129472 23679 fs_manager.cc:359] Metadata directory not provided
I20260806 02:57:21.129592 23679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:57:21.129945 23679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:57:21.133208 23679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/ts-0/data/instance:
uuid: "925263f6983345b5b3760117b9e57f33"
format_stamp: "Formatted at 2026-08-06 02:57:21 on dist-test-slave-zj8q"
server_key: "35d9bc716476534c1ab7bdea2cef1acf"
server_key_iv: "066f00ae925d75685a88b7f3152d017d"
server_key_version: "encryptionkey@0"
I20260806 02:57:21.134106 23679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/ts-0/wal/instance:
uuid: "925263f6983345b5b3760117b9e57f33"
format_stamp: "Formatted at 2026-08-06 02:57:21 on dist-test-slave-zj8q"
server_key: "35d9bc716476534c1ab7bdea2cef1acf"
server_key_iv: "066f00ae925d75685a88b7f3152d017d"
server_key_version: "encryptionkey@0"
I20260806 02:57:21.139890 23679 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 02:57:21.144018 23697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:21.145501 23679 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 02:57:21.145679 23679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/ts-0/wal
uuid: "925263f6983345b5b3760117b9e57f33"
format_stamp: "Formatted at 2026-08-06 02:57:21 on dist-test-slave-zj8q"
server_key: "35d9bc716476534c1ab7bdea2cef1acf"
server_key_iv: "066f00ae925d75685a88b7f3152d017d"
server_key_version: "encryptionkey@0"
I20260806 02:57:21.145880 23679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:57:21.170677 23679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:57:21.171538 23679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:57:21.171828 23679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:57:21.172859 23679 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:57:21.174471 23679 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:57:21.174574 23679 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:21.174669 23679 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:57:21.174727 23679 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:21.205889 23679 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.129:42527
I20260806 02:57:21.205960 23866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.129:42527 every 8 connection(s)
I20260806 02:57:21.207216 23679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/ts-0/data/info.pb
I20260806 02:57:21.217401 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 23679
I20260806 02:57:21.217569 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/ts-0/wal/instance
I20260806 02:57:21.218107 26446 external_mini_cluster.cc:1428] Setting key 1ff3965b4e5c7966309d97c006c530e5
I20260806 02:57:21.220382 23870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:21.209764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:35279 (local address 127.25.211.190:35391)
0806 02:57:21.210259 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:21.210264 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:21.211554 (+ 1290us) server_negotiation.cc:407] Connection header received
0806 02:57:21.213414 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:21.213420 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:21.213505 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:21.213603 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:21.214812 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:21.215548 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:21.217199 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:21.217694 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:21.217809 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:21.219485 (+ 1676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:21.219491 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:21.219495 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:57:21.219517 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:21.219570 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:57:21.219587 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:21.219727 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:21.219933 (+ 206us) server_negotiation.cc:299] Negotiation successful
0806 02:57:21.220031 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":229,"threads_started":1}
I20260806 02:57:21.221433 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.130:0
--local_ip_for_outbound_sockets=127.25.211.130
--webserver_interface=127.25.211.130
--webserver_port=0
--tserver_master_addrs=127.25.211.190:35391
--builtin_ntp_servers=127.25.211.148:39137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 02:57:21.221674 23871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:21.210338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:35391 (local address 127.25.211.129:35279)
0806 02:57:21.211392 (+ 1054us) negotiation.cc:107] Waiting for socket to connect
0806 02:57:21.211437 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 02:57:21.213119 (+ 1682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:57:21.213874 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:57:21.213883 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:57:21.213979 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:57:21.214632 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:21.214651 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:21.215705 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:21.215709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:21.217009 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:21.217027 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:21.217945 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:21.217953 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:21.218302 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:21.219106 (+ 804us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:57:21.219110 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:57:21.219171 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:57:21.219286 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:57:21.219717 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:57:21.219740 (+ 23us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:57:21.219746 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 02:57:21.219884 (+ 138us) client_negotiation.cc:240] Negotiation successful
0806 02:57:21.220281 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":278,"threads_started":1}
I20260806 02:57:21.223265 23867 heartbeater.cc:347] Connected to a master server at 127.25.211.190:35391
I20260806 02:57:21.223608 23867 heartbeater.cc:464] Registering TS with master...
I20260806 02:57:21.224524 23867 heartbeater.cc:511] Master 127.25.211.190:35391 requested a full tablet report, sending...
I20260806 02:57:21.226912 23583 ts_manager.cc:194] Registered new tserver with Master: 925263f6983345b5b3760117b9e57f33 (127.25.211.129:42527)
I20260806 02:57:21.228454 23583 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.211.129:35279
W20260806 02:57:21.564644 23872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:57:21.564951 23872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:57:21.565042 23872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:57:21.572825 23872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:57:21.572928 23872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:57:21.573037 23872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.130
I20260806 02:57:21.582705 23872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:39137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.211.130
--webserver_port=0
--tserver_master_addrs=127.25.211.190:35391
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.130
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:57:21.584101 23872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:57:21.585925 23872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:57:21.600033 23879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:21.600383 23878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:21.600113 23882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:57:21.601155 23872 server_base.cc:1034] running on GCE node
I20260806 02:57:21.601843 23872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:57:21.602964 23872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:57:21.604223 23872 hybrid_clock.cc:648] HybridClock initialized: now 1785985041604150 us; error 63 us; skew 500 ppm
I20260806 02:57:21.604635 23872 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:57:21.607391 23872 webserver.cc:460] Webserver started at http://127.25.211.130:35935/ using document root <none> and password file <none>
I20260806 02:57:21.608160 23872 fs_manager.cc:359] Metadata directory not provided
I20260806 02:57:21.608274 23872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:57:21.608599 23872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:57:21.611740 23872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/ts-1/data/instance:
uuid: "97a954e4b78d444c95b9113ae889a585"
format_stamp: "Formatted at 2026-08-06 02:57:21 on dist-test-slave-zj8q"
server_key: "1a639ddf85a105aed55ae30c20572d88"
server_key_iv: "000417059551cf0de8f6e27e5cf2f902"
server_key_version: "encryptionkey@0"
I20260806 02:57:21.612604 23872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/ts-1/wal/instance:
uuid: "97a954e4b78d444c95b9113ae889a585"
format_stamp: "Formatted at 2026-08-06 02:57:21 on dist-test-slave-zj8q"
server_key: "1a639ddf85a105aed55ae30c20572d88"
server_key_iv: "000417059551cf0de8f6e27e5cf2f902"
server_key_version: "encryptionkey@0"
I20260806 02:57:21.618638 23872 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260806 02:57:21.622720 23895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:21.624362 23872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 02:57:21.624550 23872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/ts-1/wal
uuid: "97a954e4b78d444c95b9113ae889a585"
format_stamp: "Formatted at 2026-08-06 02:57:21 on dist-test-slave-zj8q"
server_key: "1a639ddf85a105aed55ae30c20572d88"
server_key_iv: "000417059551cf0de8f6e27e5cf2f902"
server_key_version: "encryptionkey@0"
I20260806 02:57:21.624781 23872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:57:21.654786 23872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:57:21.655699 23872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:57:21.655997 23872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:57:21.657090 23872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:57:21.658806 23872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:57:21.658916 23872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:21.659030 23872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:57:21.659099 23872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:21.691268 23872 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.130:37771
I20260806 02:57:21.691388 24062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.130:37771 every 8 connection(s)
I20260806 02:57:21.692642 23872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/ts-1/data/info.pb
I20260806 02:57:21.695369 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 23872
I20260806 02:57:21.695550 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/ts-1/wal/instance
I20260806 02:57:21.695989 26446 external_mini_cluster.cc:1428] Setting key 3049b7f5af8b2f84ff70c9260a7d07a2
I20260806 02:57:21.698920 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.211.131:0
--local_ip_for_outbound_sockets=127.25.211.131
--webserver_interface=127.25.211.131
--webserver_port=0
--tserver_master_addrs=127.25.211.190:35391
--builtin_ntp_servers=127.25.211.148:39137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 02:57:21.706121 23870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:21.695316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.130:56097 (local address 127.25.211.190:35391)
0806 02:57:21.695533 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:21.695540 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:21.696887 (+ 1347us) server_negotiation.cc:407] Connection header received
0806 02:57:21.698921 (+ 2034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:21.698927 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:21.699013 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:21.699123 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:21.700616 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:21.701370 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:21.703451 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:21.703890 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:21.703987 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:21.705428 (+ 1441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:21.705433 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:21.705436 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:57:21.705454 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:21.705487 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:57:21.705499 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:21.705578 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:21.705804 (+ 226us) server_negotiation.cc:299] Negotiation successful
0806 02:57:21.705883 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 02:57:21.707482 24067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:21.695726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:35391 (local address 127.25.211.130:56097)
0806 02:57:21.696731 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0806 02:57:21.696775 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 02:57:21.698660 (+ 1885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:57:21.699536 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:57:21.699546 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:57:21.699660 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:57:21.700432 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:21.700464 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:21.701526 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:21.701531 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:21.703296 (+ 1765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:21.703311 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:21.704102 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:21.704108 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:21.704386 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:21.705081 (+ 695us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:57:21.705085 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:57:21.705142 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:57:21.705233 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:57:21.705601 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:57:21.705611 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:57:21.705618 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 02:57:21.705791 (+ 173us) client_negotiation.cc:240] Negotiation successful
0806 02:57:21.706202 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":308,"threads_started":1}
I20260806 02:57:21.708920 24063 heartbeater.cc:347] Connected to a master server at 127.25.211.190:35391
I20260806 02:57:21.709203 24063 heartbeater.cc:464] Registering TS with master...
I20260806 02:57:21.709936 24063 heartbeater.cc:511] Master 127.25.211.190:35391 requested a full tablet report, sending...
I20260806 02:57:21.711428 23583 ts_manager.cc:194] Registered new tserver with Master: 97a954e4b78d444c95b9113ae889a585 (127.25.211.130:37771)
I20260806 02:57:21.712447 23583 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.211.130:56097
W20260806 02:57:22.049897 24069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:57:22.050278 24069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:57:22.050415 24069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:57:22.057857 24069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:57:22.057968 24069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:57:22.058117 24069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.211.131
I20260806 02:57:22.067768 24069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:39137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.211.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.211.131
--webserver_port=0
--tserver_master_addrs=127.25.211.190:35391
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.211.131
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:57:22.069079 24069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:57:22.070902 24069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:57:22.084466 24076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:22.084378 24079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:22.084436 24077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:57:22.085843 24069 server_base.cc:1034] running on GCE node
I20260806 02:57:22.086586 24069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:57:22.087675 24069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:57:22.088896 24069 hybrid_clock.cc:648] HybridClock initialized: now 1785985042088836 us; error 51 us; skew 500 ppm
I20260806 02:57:22.089262 24069 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:57:22.091643 24069 webserver.cc:460] Webserver started at http://127.25.211.131:35073/ using document root <none> and password file <none>
I20260806 02:57:22.092355 24069 fs_manager.cc:359] Metadata directory not provided
I20260806 02:57:22.092487 24069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:57:22.092794 24069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:57:22.095815 24069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/ts-2/data/instance:
uuid: "46dcc2b09e12484e8e53004104b71230"
format_stamp: "Formatted at 2026-08-06 02:57:22 on dist-test-slave-zj8q"
server_key: "1aa8abcb0d94791581c75e3dfe752978"
server_key_iv: "1a458e6ac5b0f071f61cdc279f977160"
server_key_version: "encryptionkey@0"
I20260806 02:57:22.096640 24069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/ts-2/wal/instance:
uuid: "46dcc2b09e12484e8e53004104b71230"
format_stamp: "Formatted at 2026-08-06 02:57:22 on dist-test-slave-zj8q"
server_key: "1aa8abcb0d94791581c75e3dfe752978"
server_key_iv: "1a458e6ac5b0f071f61cdc279f977160"
server_key_version: "encryptionkey@0"
I20260806 02:57:22.102092 24069 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260806 02:57:22.105962 24091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:22.107441 24069 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:57:22.107626 24069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/ts-2/wal
uuid: "46dcc2b09e12484e8e53004104b71230"
format_stamp: "Formatted at 2026-08-06 02:57:22 on dist-test-slave-zj8q"
server_key: "1aa8abcb0d94791581c75e3dfe752978"
server_key_iv: "1a458e6ac5b0f071f61cdc279f977160"
server_key_version: "encryptionkey@0"
I20260806 02:57:22.107852 24069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:57:22.127624 24069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:57:22.128516 24069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:57:22.128822 24069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:57:22.129959 24069 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 02:57:22.131618 24069 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 02:57:22.131719 24069 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:22.131829 24069 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 02:57:22.131902 24069 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:22.163125 24069 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.131:38803
I20260806 02:57:22.163142 24252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.131:38803 every 8 connection(s)
I20260806 02:57:22.164449 24069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/ts-2/data/info.pb
I20260806 02:57:22.174325 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 24069
I20260806 02:57:22.174506 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/ts-2/wal/instance
I20260806 02:57:22.175058 26446 external_mini_cluster.cc:1428] Setting key 308281e127be533fabed7417d45f0352
I20260806 02:57:22.178108 23870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:22.167353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.131:44295 (local address 127.25.211.190:35391)
0806 02:57:22.167600 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:22.167608 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:22.168998 (+ 1390us) server_negotiation.cc:407] Connection header received
0806 02:57:22.170949 (+ 1951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:22.170954 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:22.171034 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:22.171136 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:22.172259 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:22.173037 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:22.174873 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:22.175501 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:22.175645 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:22.177280 (+ 1635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:22.177287 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:22.177291 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:57:22.177313 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:22.177354 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:57:22.177370 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:22.177471 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:22.177684 (+ 213us) server_negotiation.cc:299] Negotiation successful
0806 02:57:22.177792 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260806 02:57:22.179209 24257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:22.167853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:35391 (local address 127.25.211.131:44295)
0806 02:57:22.168824 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0806 02:57:22.168864 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 02:57:22.170633 (+ 1769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:57:22.171343 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:57:22.171352 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:57:22.171458 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:57:22.172102 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:22.172122 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:22.173175 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:22.173179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:22.174704 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:22.174721 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:22.175781 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:22.175790 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:22.176029 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:22.176907 (+ 878us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 02:57:22.176910 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:57:22.176974 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 02:57:22.177085 (+ 111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:57:22.177460 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:57:22.177467 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:57:22.177473 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 02:57:22.177618 (+ 145us) client_negotiation.cc:240] Negotiation successful
0806 02:57:22.178044 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":285,"threads_started":1}
I20260806 02:57:22.180667 24254 heartbeater.cc:347] Connected to a master server at 127.25.211.190:35391
I20260806 02:57:22.181010 24254 heartbeater.cc:464] Registering TS with master...
I20260806 02:57:22.181836 24254 heartbeater.cc:511] Master 127.25.211.190:35391 requested a full tablet report, sending...
I20260806 02:57:22.183483 23583 ts_manager.cc:194] Registered new tserver with Master: 46dcc2b09e12484e8e53004104b71230 (127.25.211.131:38803)
I20260806 02:57:22.184334 23583 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.211.131:44295
I20260806 02:57:22.190968 26446 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 02:57:22.201524 23870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:22.195499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38162 (local address 127.25.211.190:35391)
0806 02:57:22.195687 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:22.195691 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:22.196131 (+ 440us) server_negotiation.cc:407] Connection header received
0806 02:57:22.196285 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:22.196289 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:22.196354 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:22.196444 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:22.196880 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:22.197565 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:22.199312 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:22.199782 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:22.199887 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:22.200908 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:22.200913 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:22.200917 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:57:22.200937 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:22.200974 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:57:22.200987 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:22.201069 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:22.201189 (+ 120us) server_negotiation.cc:299] Negotiation successful
0806 02:57:22.201274 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 02:57:22.205852 23583 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38162:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 02:57:22.212939 23583 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 02:57:22.232254 23867 heartbeater.cc:503] Master 127.25.211.190:35391 was elected leader, sending a full tablet report...
I20260806 02:57:22.249537 24270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:22.224770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.131:38803 (local address 127.0.0.1:48300)
0806 02:57:22.226434 (+ 1664us) negotiation.cc:107] Waiting for socket to connect
0806 02:57:22.226485 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 02:57:22.226650 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:57:22.241792 (+ 15142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:57:22.241800 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:57:22.241841 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:57:22.242422 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:22.242434 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:22.244241 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:22.244247 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:22.247082 (+ 2835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:22.247095 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:22.248466 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:22.248473 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:22.248668 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:22.248734 (+ 66us) client_negotiation.cc:769] Sending connection context
0806 02:57:22.248857 (+ 123us) client_negotiation.cc:240] Negotiation successful
0806 02:57:22.249069 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1354,"mutex_wait_us":240,"spinlock_wait_cycles":1408,"thread_start_us":15695,"threads_started":1}
I20260806 02:57:22.250069 24275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:22.227186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39410 (local address 127.25.211.130:37771)
0806 02:57:22.241649 (+ 14463us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:22.241656 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:22.241697 (+ 41us) server_negotiation.cc:407] Connection header received
0806 02:57:22.241803 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:22.241810 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:22.242019 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:22.242238 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:57:22.243271 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:22.245021 (+ 1750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:22.247018 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:22.248403 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:22.248565 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:22.248769 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:22.249535 (+ 766us) server_negotiation.cc:299] Negotiation successful
0806 02:57:22.249721 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14308,"thread_start_us":14176,"threads_started":1}
I20260806 02:57:22.250136 24271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:22.225018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48300 (local address 127.25.211.131:38803)
0806 02:57:22.241192 (+ 16174us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:22.241202 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:22.241240 (+ 38us) server_negotiation.cc:407] Connection header received
0806 02:57:22.241343 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:22.241350 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:22.241578 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:22.241770 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:57:22.242632 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:22.244071 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:22.247336 (+ 3265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:22.248275 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:22.248466 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:22.248708 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:22.249591 (+ 883us) server_negotiation.cc:299] Negotiation successful
0806 02:57:22.249789 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15975,"spinlock_wait_cycles":27520,"thread_start_us":15954,"threads_started":1}
I20260806 02:57:22.250901 24269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:22.226842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.130:37771 (local address 127.0.0.1:39410)
0806 02:57:22.240747 (+ 13905us) negotiation.cc:107] Waiting for socket to connect
0806 02:57:22.240767 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 02:57:22.240900 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:57:22.242272 (+ 1372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:57:22.242278 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:57:22.242305 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:57:22.242810 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:22.242823 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:22.245160 (+ 2337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:22.245165 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:22.246823 (+ 1658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:22.246836 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:22.248911 (+ 2075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:22.248918 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:22.249104 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:22.249377 (+ 273us) client_negotiation.cc:769] Sending connection context
0806 02:57:22.249465 (+ 88us) client_negotiation.cc:240] Negotiation successful
0806 02:57:22.250558 (+ 1093us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13773,"mutex_wait_us":51,"thread_start_us":14712,"threads_started":1}
I20260806 02:57:22.251267 24274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:22.224969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.129:42527 (local address 127.0.0.1:38698)
0806 02:57:22.240231 (+ 15262us) negotiation.cc:107] Waiting for socket to connect
0806 02:57:22.240254 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 02:57:22.240423 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:57:22.241695 (+ 1272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:57:22.241703 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:57:22.241736 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:57:22.242371 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:22.242397 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:22.244420 (+ 2023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:22.244425 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:22.247080 (+ 2655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:22.247095 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:22.250440 (+ 3345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:22.250447 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:22.250594 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:22.250616 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 02:57:22.250686 (+ 70us) client_negotiation.cc:240] Negotiation successful
0806 02:57:22.250818 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15112,"mutex_wait_us":156,"spinlock_wait_cycles":1792,"thread_start_us":1014,"threads_started":1}
I20260806 02:57:22.251302 24273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:22.225793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38698 (local address 127.25.211.129:42527)
0806 02:57:22.239090 (+ 13297us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:22.239102 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:22.240597 (+ 1495us) server_negotiation.cc:407] Connection header received
0806 02:57:22.240749 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:22.240760 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:22.241113 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:22.241346 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:57:22.242619 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:22.244266 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:22.247315 (+ 3049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:22.248757 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:22.248960 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:22.249330 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:22.250781 (+ 1451us) server_negotiation.cc:299] Negotiation successful
0806 02:57:22.250953 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13044,"thread_start_us":13079,"threads_started":1}
I20260806 02:57:22.254577 23970 tablet_service.cc:1467] Processing CreateTablet for tablet 483a9443000f4adbbab62a2a3710bcb2 (DEFAULT_TABLE table=test-table [id=a131d021eddc43ff9f578c5256f43d3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:57:22.254617 24161 tablet_service.cc:1467] Processing CreateTablet for tablet 483a9443000f4adbbab62a2a3710bcb2 (DEFAULT_TABLE table=test-table [id=a131d021eddc43ff9f578c5256f43d3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:57:22.254920 23766 tablet_service.cc:1467] Processing CreateTablet for tablet 483a9443000f4adbbab62a2a3710bcb2 (DEFAULT_TABLE table=test-table [id=a131d021eddc43ff9f578c5256f43d3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 02:57:22.255924 23970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 483a9443000f4adbbab62a2a3710bcb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:57:22.256125 24161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 483a9443000f4adbbab62a2a3710bcb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:57:22.256316 23766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 483a9443000f4adbbab62a2a3710bcb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:57:22.267339 24276 tablet_bootstrap.cc:492] T 483a9443000f4adbbab62a2a3710bcb2 P 97a954e4b78d444c95b9113ae889a585: Bootstrap starting.
I20260806 02:57:22.270437 24277 tablet_bootstrap.cc:492] T 483a9443000f4adbbab62a2a3710bcb2 P 46dcc2b09e12484e8e53004104b71230: Bootstrap starting.
I20260806 02:57:22.270740 24276 tablet_bootstrap.cc:654] T 483a9443000f4adbbab62a2a3710bcb2 P 97a954e4b78d444c95b9113ae889a585: Neither blocks nor log segments found. Creating new log.
I20260806 02:57:22.272055 24276 log.cc:824] T 483a9443000f4adbbab62a2a3710bcb2 P 97a954e4b78d444c95b9113ae889a585: Log is configured to *not* fsync() on all Append() calls
I20260806 02:57:22.272110 24278 tablet_bootstrap.cc:492] T 483a9443000f4adbbab62a2a3710bcb2 P 925263f6983345b5b3760117b9e57f33: Bootstrap starting.
I20260806 02:57:22.273933 24277 tablet_bootstrap.cc:654] T 483a9443000f4adbbab62a2a3710bcb2 P 46dcc2b09e12484e8e53004104b71230: Neither blocks nor log segments found. Creating new log.
I20260806 02:57:22.275221 24277 log.cc:824] T 483a9443000f4adbbab62a2a3710bcb2 P 46dcc2b09e12484e8e53004104b71230: Log is configured to *not* fsync() on all Append() calls
I20260806 02:57:22.275288 24278 tablet_bootstrap.cc:654] T 483a9443000f4adbbab62a2a3710bcb2 P 925263f6983345b5b3760117b9e57f33: Neither blocks nor log segments found. Creating new log.
I20260806 02:57:22.275731 24276 tablet_bootstrap.cc:492] T 483a9443000f4adbbab62a2a3710bcb2 P 97a954e4b78d444c95b9113ae889a585: No bootstrap required, opened a new log
I20260806 02:57:22.276068 24276 ts_tablet_manager.cc:1397] T 483a9443000f4adbbab62a2a3710bcb2 P 97a954e4b78d444c95b9113ae889a585: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260806 02:57:22.276425 24278 log.cc:824] T 483a9443000f4adbbab62a2a3710bcb2 P 925263f6983345b5b3760117b9e57f33: Log is configured to *not* fsync() on all Append() calls
I20260806 02:57:22.278641 24277 tablet_bootstrap.cc:492] T 483a9443000f4adbbab62a2a3710bcb2 P 46dcc2b09e12484e8e53004104b71230: No bootstrap required, opened a new log
I20260806 02:57:22.278980 24277 ts_tablet_manager.cc:1397] T 483a9443000f4adbbab62a2a3710bcb2 P 46dcc2b09e12484e8e53004104b71230: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260806 02:57:22.283735 24276 raft_consensus.cc:348] T 483a9443000f4adbbab62a2a3710bcb2 P 97a954e4b78d444c95b9113ae889a585 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46dcc2b09e12484e8e53004104b71230" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38803 } } peers { permanent_uuid: "925263f6983345b5b3760117b9e57f33" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42527 } } peers { permanent_uuid: "97a954e4b78d444c95b9113ae889a585" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 37771 } }
I20260806 02:57:22.284119 24276 raft_consensus.cc:374] T 483a9443000f4adbbab62a2a3710bcb2 P 97a954e4b78d444c95b9113ae889a585 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:57:22.284281 24276 raft_consensus.cc:729] T 483a9443000f4adbbab62a2a3710bcb2 P 97a954e4b78d444c95b9113ae889a585 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97a954e4b78d444c95b9113ae889a585, State: Initialized, Role: FOLLOWER
I20260806 02:57:22.285024 24276 consensus_queue.cc:260] T 483a9443000f4adbbab62a2a3710bcb2 P 97a954e4b78d444c95b9113ae889a585 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46dcc2b09e12484e8e53004104b71230" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38803 } } peers { permanent_uuid: "925263f6983345b5b3760117b9e57f33" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42527 } } peers { permanent_uuid: "97a954e4b78d444c95b9113ae889a585" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 37771 } }
I20260806 02:57:22.286294 24063 heartbeater.cc:503] Master 127.25.211.190:35391 was elected leader, sending a full tablet report...
I20260806 02:57:22.286994 24276 ts_tablet_manager.cc:1428] T 483a9443000f4adbbab62a2a3710bcb2 P 97a954e4b78d444c95b9113ae889a585: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260806 02:57:22.287216 24278 tablet_bootstrap.cc:492] T 483a9443000f4adbbab62a2a3710bcb2 P 925263f6983345b5b3760117b9e57f33: No bootstrap required, opened a new log
I20260806 02:57:22.287508 24278 ts_tablet_manager.cc:1397] T 483a9443000f4adbbab62a2a3710bcb2 P 925263f6983345b5b3760117b9e57f33: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.004s
I20260806 02:57:22.289400 24277 raft_consensus.cc:348] T 483a9443000f4adbbab62a2a3710bcb2 P 46dcc2b09e12484e8e53004104b71230 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46dcc2b09e12484e8e53004104b71230" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38803 } } peers { permanent_uuid: "925263f6983345b5b3760117b9e57f33" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42527 } } peers { permanent_uuid: "97a954e4b78d444c95b9113ae889a585" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 37771 } }
I20260806 02:57:22.289919 24277 raft_consensus.cc:374] T 483a9443000f4adbbab62a2a3710bcb2 P 46dcc2b09e12484e8e53004104b71230 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:57:22.290061 24277 raft_consensus.cc:729] T 483a9443000f4adbbab62a2a3710bcb2 P 46dcc2b09e12484e8e53004104b71230 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46dcc2b09e12484e8e53004104b71230, State: Initialized, Role: FOLLOWER
I20260806 02:57:22.290792 24277 consensus_queue.cc:260] T 483a9443000f4adbbab62a2a3710bcb2 P 46dcc2b09e12484e8e53004104b71230 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46dcc2b09e12484e8e53004104b71230" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38803 } } peers { permanent_uuid: "925263f6983345b5b3760117b9e57f33" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42527 } } peers { permanent_uuid: "97a954e4b78d444c95b9113ae889a585" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 37771 } }
I20260806 02:57:22.292029 24254 heartbeater.cc:503] Master 127.25.211.190:35391 was elected leader, sending a full tablet report...
I20260806 02:57:22.292835 24277 ts_tablet_manager.cc:1428] T 483a9443000f4adbbab62a2a3710bcb2 P 46dcc2b09e12484e8e53004104b71230: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260806 02:57:22.296382 24278 raft_consensus.cc:348] T 483a9443000f4adbbab62a2a3710bcb2 P 925263f6983345b5b3760117b9e57f33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46dcc2b09e12484e8e53004104b71230" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38803 } } peers { permanent_uuid: "925263f6983345b5b3760117b9e57f33" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42527 } } peers { permanent_uuid: "97a954e4b78d444c95b9113ae889a585" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 37771 } }
I20260806 02:57:22.296768 24278 raft_consensus.cc:374] T 483a9443000f4adbbab62a2a3710bcb2 P 925263f6983345b5b3760117b9e57f33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:57:22.296865 24278 raft_consensus.cc:729] T 483a9443000f4adbbab62a2a3710bcb2 P 925263f6983345b5b3760117b9e57f33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 925263f6983345b5b3760117b9e57f33, State: Initialized, Role: FOLLOWER
I20260806 02:57:22.297425 24278 consensus_queue.cc:260] T 483a9443000f4adbbab62a2a3710bcb2 P 925263f6983345b5b3760117b9e57f33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46dcc2b09e12484e8e53004104b71230" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38803 } } peers { permanent_uuid: "925263f6983345b5b3760117b9e57f33" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42527 } } peers { permanent_uuid: "97a954e4b78d444c95b9113ae889a585" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 37771 } }
I20260806 02:57:22.299098 24278 ts_tablet_manager.cc:1428] T 483a9443000f4adbbab62a2a3710bcb2 P 925263f6983345b5b3760117b9e57f33: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
W20260806 02:57:22.416282 24256 tablet.cc:2369] T 483a9443000f4adbbab62a2a3710bcb2 P 46dcc2b09e12484e8e53004104b71230: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:57:22.444720 24065 tablet.cc:2369] T 483a9443000f4adbbab62a2a3710bcb2 P 97a954e4b78d444c95b9113ae889a585: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 02:57:22.459641 23869 tablet.cc:2369] T 483a9443000f4adbbab62a2a3710bcb2 P 925263f6983345b5b3760117b9e57f33: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:57:22.512537 24288 raft_consensus.cc:482] T 483a9443000f4adbbab62a2a3710bcb2 P 925263f6983345b5b3760117b9e57f33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 02:57:22.512941 24288 raft_consensus.cc:504] T 483a9443000f4adbbab62a2a3710bcb2 P 925263f6983345b5b3760117b9e57f33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46dcc2b09e12484e8e53004104b71230" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38803 } } peers { permanent_uuid: "925263f6983345b5b3760117b9e57f33" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42527 } } peers { permanent_uuid: "97a954e4b78d444c95b9113ae889a585" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 37771 } }
I20260806 02:57:22.514889 24288 leader_election.cc:288] T 483a9443000f4adbbab62a2a3710bcb2 P 925263f6983345b5b3760117b9e57f33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46dcc2b09e12484e8e53004104b71230 (127.25.211.131:38803), 97a954e4b78d444c95b9113ae889a585 (127.25.211.130:37771)
I20260806 02:57:22.523280 24275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:22.515618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:43145 (local address 127.25.211.130:37771)
0806 02:57:22.516057 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:22.516063 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:22.516482 (+ 419us) server_negotiation.cc:407] Connection header received
0806 02:57:22.516659 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:22.516666 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:22.516743 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:22.516811 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:57:22.517415 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:22.518439 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:22.521296 (+ 2857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:22.522278 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:22.522423 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:22.522478 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:22.522861 (+ 383us) server_negotiation.cc:299] Negotiation successful
0806 02:57:22.522993 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299}
I20260806 02:57:22.523324 24271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:22.515436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.25.211.129:53861 (local address 127.25.211.131:38803)
0806 02:57:22.515880 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:22.515889 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:22.516154 (+ 265us) server_negotiation.cc:407] Connection header received
0806 02:57:22.516415 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:22.516419 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:22.516482 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:22.516549 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 02:57:22.517129 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:22.518262 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:22.521340 (+ 3078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:22.522451 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:22.522610 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:22.522677 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:22.522976 (+ 299us) server_negotiation.cc:299] Negotiation successful
0806 02:57:22.523110 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265}
I20260806 02:57:22.523350 24291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:22.515435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.130:37771 (local address 127.25.211.129:43145)
0806 02:57:22.516382 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0806 02:57:22.516397 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:57:22.516505 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:57:22.516874 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:57:22.516879 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:57:22.516906 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:57:22.517233 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:22.517242 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:22.518568 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:22.518573 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:22.521042 (+ 2469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:22.521066 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:22.522502 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:22.522509 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:22.522677 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:22.522712 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 02:57:22.522797 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 02:57:22.522911 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":825,"mutex_wait_us":77,"thread_start_us":593,"threads_started":1}
I20260806 02:57:22.523803 24289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:22.515216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.131:38803 (local address 127.25.211.129:53861)
0806 02:57:22.516025 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0806 02:57:22.516046 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 02:57:22.516239 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:57:22.516576 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:57:22.516580 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:57:22.516604 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 02:57:22.516941 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:22.516958 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:22.518397 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:22.518401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:22.521041 (+ 2640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:57:22.521066 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:57:22.522665 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:57:22.522670 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:57:22.522802 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:22.522835 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 02:57:22.522926 (+ 91us) client_negotiation.cc:240] Negotiation successful
0806 02:57:22.523051 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"mutex_wait_us":118,"thread_start_us":448,"threads_started":1}
I20260806 02:57:22.524436 24001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "483a9443000f4adbbab62a2a3710bcb2" candidate_uuid: "925263f6983345b5b3760117b9e57f33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97a954e4b78d444c95b9113ae889a585" is_pre_election: true
I20260806 02:57:22.524698 24188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "483a9443000f4adbbab62a2a3710bcb2" candidate_uuid: "925263f6983345b5b3760117b9e57f33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46dcc2b09e12484e8e53004104b71230" is_pre_election: true
I20260806 02:57:22.524962 24001 raft_consensus.cc:2454] T 483a9443000f4adbbab62a2a3710bcb2 P 97a954e4b78d444c95b9113ae889a585 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 925263f6983345b5b3760117b9e57f33 in term 0.
I20260806 02:57:22.525185 24188 raft_consensus.cc:2454] T 483a9443000f4adbbab62a2a3710bcb2 P 46dcc2b09e12484e8e53004104b71230 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 925263f6983345b5b3760117b9e57f33 in term 0.
I20260806 02:57:22.525664 23700 leader_election.cc:302] T 483a9443000f4adbbab62a2a3710bcb2 P 925263f6983345b5b3760117b9e57f33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 925263f6983345b5b3760117b9e57f33, 97a954e4b78d444c95b9113ae889a585; no voters:
I20260806 02:57:22.526139 24288 raft_consensus.cc:2781] T 483a9443000f4adbbab62a2a3710bcb2 P 925263f6983345b5b3760117b9e57f33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 02:57:22.526336 24288 raft_consensus.cc:482] T 483a9443000f4adbbab62a2a3710bcb2 P 925263f6983345b5b3760117b9e57f33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 02:57:22.526437 24288 raft_consensus.cc:3037] T 483a9443000f4adbbab62a2a3710bcb2 P 925263f6983345b5b3760117b9e57f33 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:57:22.528880 24288 raft_consensus.cc:504] T 483a9443000f4adbbab62a2a3710bcb2 P 925263f6983345b5b3760117b9e57f33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46dcc2b09e12484e8e53004104b71230" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38803 } } peers { permanent_uuid: "925263f6983345b5b3760117b9e57f33" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42527 } } peers { permanent_uuid: "97a954e4b78d444c95b9113ae889a585" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 37771 } }
I20260806 02:57:22.529543 24288 leader_election.cc:288] T 483a9443000f4adbbab62a2a3710bcb2 P 925263f6983345b5b3760117b9e57f33 [CANDIDATE]: Term 1 election: Requested vote from peers 46dcc2b09e12484e8e53004104b71230 (127.25.211.131:38803), 97a954e4b78d444c95b9113ae889a585 (127.25.211.130:37771)
I20260806 02:57:22.530014 24188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "483a9443000f4adbbab62a2a3710bcb2" candidate_uuid: "925263f6983345b5b3760117b9e57f33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46dcc2b09e12484e8e53004104b71230"
I20260806 02:57:22.530047 24001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "483a9443000f4adbbab62a2a3710bcb2" candidate_uuid: "925263f6983345b5b3760117b9e57f33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97a954e4b78d444c95b9113ae889a585"
I20260806 02:57:22.530252 24188 raft_consensus.cc:3037] T 483a9443000f4adbbab62a2a3710bcb2 P 46dcc2b09e12484e8e53004104b71230 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:57:22.530287 24001 raft_consensus.cc:3037] T 483a9443000f4adbbab62a2a3710bcb2 P 97a954e4b78d444c95b9113ae889a585 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:57:22.532516 24001 raft_consensus.cc:2454] T 483a9443000f4adbbab62a2a3710bcb2 P 97a954e4b78d444c95b9113ae889a585 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 925263f6983345b5b3760117b9e57f33 in term 1.
I20260806 02:57:22.532516 24188 raft_consensus.cc:2454] T 483a9443000f4adbbab62a2a3710bcb2 P 46dcc2b09e12484e8e53004104b71230 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 925263f6983345b5b3760117b9e57f33 in term 1.
I20260806 02:57:22.532981 23701 leader_election.cc:302] T 483a9443000f4adbbab62a2a3710bcb2 P 925263f6983345b5b3760117b9e57f33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46dcc2b09e12484e8e53004104b71230, 925263f6983345b5b3760117b9e57f33; no voters:
I20260806 02:57:22.533317 24288 raft_consensus.cc:2781] T 483a9443000f4adbbab62a2a3710bcb2 P 925263f6983345b5b3760117b9e57f33 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:57:22.533776 24288 raft_consensus.cc:686] T 483a9443000f4adbbab62a2a3710bcb2 P 925263f6983345b5b3760117b9e57f33 [term 1 LEADER]: Becoming Leader. State: Replica: 925263f6983345b5b3760117b9e57f33, State: Running, Role: LEADER
I20260806 02:57:22.534317 24288 consensus_queue.cc:237] T 483a9443000f4adbbab62a2a3710bcb2 P 925263f6983345b5b3760117b9e57f33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46dcc2b09e12484e8e53004104b71230" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38803 } } peers { permanent_uuid: "925263f6983345b5b3760117b9e57f33" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42527 } } peers { permanent_uuid: "97a954e4b78d444c95b9113ae889a585" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 37771 } }
I20260806 02:57:22.539511 23582 catalog_manager.cc:5495] T 483a9443000f4adbbab62a2a3710bcb2 P 925263f6983345b5b3760117b9e57f33 reported cstate change: term changed from 0 to 1, leader changed from <none> to 925263f6983345b5b3760117b9e57f33 (127.25.211.129). New cstate: current_term: 1 leader_uuid: "925263f6983345b5b3760117b9e57f33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46dcc2b09e12484e8e53004104b71230" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "925263f6983345b5b3760117b9e57f33" member_type: VOTER last_known_addr { host: "127.25.211.129" port: 42527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97a954e4b78d444c95b9113ae889a585" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 37771 } health_report { overall_health: UNKNOWN } } }
I20260806 02:57:22.606719 24273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:22.600241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38712 (local address 127.25.211.129:42527)
0806 02:57:22.600458 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:22.600465 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:22.600509 (+ 44us) server_negotiation.cc:407] Connection header received
0806 02:57:22.600725 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:22.600731 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:22.600857 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:22.601024 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:57:22.601316 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:22.602445 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:22.603865 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:22.604560 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:22.604696 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:22.605472 (+ 776us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:57:22.606050 (+ 578us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:57:22.606242 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:22.606361 (+ 119us) server_negotiation.cc:299] Negotiation successful
0806 02:57:22.606456 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260806 02:57:22.616712 24001 raft_consensus.cc:1261] T 483a9443000f4adbbab62a2a3710bcb2 P 97a954e4b78d444c95b9113ae889a585 [term 1 FOLLOWER]: Refusing update from remote peer 925263f6983345b5b3760117b9e57f33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:57:22.616711 24188 raft_consensus.cc:1261] T 483a9443000f4adbbab62a2a3710bcb2 P 46dcc2b09e12484e8e53004104b71230 [term 1 FOLLOWER]: Refusing update from remote peer 925263f6983345b5b3760117b9e57f33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:57:22.617839 24292 consensus_queue.cc:1035] T 483a9443000f4adbbab62a2a3710bcb2 P 925263f6983345b5b3760117b9e57f33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97a954e4b78d444c95b9113ae889a585" member_type: VOTER last_known_addr { host: "127.25.211.130" port: 37771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:57:22.618249 24288 consensus_queue.cc:1035] T 483a9443000f4adbbab62a2a3710bcb2 P 925263f6983345b5b3760117b9e57f33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46dcc2b09e12484e8e53004104b71230" member_type: VOTER last_known_addr { host: "127.25.211.131" port: 38803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:57:22.631611 24295 mvcc.cc:205] Tried to move back new op lower bound from 7315394734546841600 to 7315394734224666624. Current Snapshot: MvccSnapshot[applied={T|T < 7315394734546841600}]
I20260806 02:57:22.632256 24296 mvcc.cc:205] Tried to move back new op lower bound from 7315394734546841600 to 7315394734224666624. Current Snapshot: MvccSnapshot[applied={T|T < 7315394734546841600}]
I20260806 02:57:22.638299 24299 mvcc.cc:205] Tried to move back new op lower bound from 7315394734546841600 to 7315394734224666624. Current Snapshot: MvccSnapshot[applied={T|T < 7315394734546841600}]
I20260806 02:57:22.646023 23582 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:57:22.646358 23582 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 02:57:22.650344 23582 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e92c1dc693d549c797c36c7f08faf276: Sending DeleteTablet for 3 replicas of tablet 483a9443000f4adbbab62a2a3710bcb2
I20260806 02:57:22.651607 24161 tablet_service.cc:1514] Processing DeleteTablet for tablet 483a9443000f4adbbab62a2a3710bcb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:57:22 UTC) from {username='slave'} at 127.0.0.1:48300
I20260806 02:57:22.651840 23766 tablet_service.cc:1514] Processing DeleteTablet for tablet 483a9443000f4adbbab62a2a3710bcb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:57:22 UTC) from {username='slave'} at 127.0.0.1:38698
I20260806 02:57:22.652622 24311 tablet_replica.cc:330] T 483a9443000f4adbbab62a2a3710bcb2 P 46dcc2b09e12484e8e53004104b71230: stopping tablet replica
I20260806 02:57:22.652566 23970 tablet_service.cc:1514] Processing DeleteTablet for tablet 483a9443000f4adbbab62a2a3710bcb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 02:57:22 UTC) from {username='slave'} at 127.0.0.1:39410
I20260806 02:57:22.652856 24313 tablet_replica.cc:330] T 483a9443000f4adbbab62a2a3710bcb2 P 925263f6983345b5b3760117b9e57f33: stopping tablet replica
I20260806 02:57:22.653327 24311 raft_consensus.cc:2229] T 483a9443000f4adbbab62a2a3710bcb2 P 46dcc2b09e12484e8e53004104b71230 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:57:22.653332 24313 raft_consensus.cc:2229] T 483a9443000f4adbbab62a2a3710bcb2 P 925263f6983345b5b3760117b9e57f33 [term 1 LEADER]: Raft consensus shutting down.
I20260806 02:57:22.653923 24311 raft_consensus.cc:2258] T 483a9443000f4adbbab62a2a3710bcb2 P 46dcc2b09e12484e8e53004104b71230 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:57:22.654178 24313 raft_consensus.cc:2258] T 483a9443000f4adbbab62a2a3710bcb2 P 925263f6983345b5b3760117b9e57f33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:57:22.655022 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 23679
I20260806 02:57:22.657272 24313 ts_tablet_manager.cc:1905] T 483a9443000f4adbbab62a2a3710bcb2 P 925263f6983345b5b3760117b9e57f33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:57:22.665674 24311 ts_tablet_manager.cc:1905] T 483a9443000f4adbbab62a2a3710bcb2 P 46dcc2b09e12484e8e53004104b71230: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:57:22.665750 24313 ts_tablet_manager.cc:1918] T 483a9443000f4adbbab62a2a3710bcb2 P 925263f6983345b5b3760117b9e57f33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:57:22.665923 24313 log.cc:1196] T 483a9443000f4adbbab62a2a3710bcb2 P 925263f6983345b5b3760117b9e57f33: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/ts-0/wal/wals/483a9443000f4adbbab62a2a3710bcb2
I20260806 02:57:22.666504 24313 ts_tablet_manager.cc:1939] T 483a9443000f4adbbab62a2a3710bcb2 P 925263f6983345b5b3760117b9e57f33: Deleting consensus metadata
I20260806 02:57:22.668413 23564 catalog_manager.cc:4841] TS 925263f6983345b5b3760117b9e57f33 (127.25.211.129:42527): tablet 483a9443000f4adbbab62a2a3710bcb2 (table test-table [id=a131d021eddc43ff9f578c5256f43d3a]) successfully deleted
I20260806 02:57:22.671443 24311 ts_tablet_manager.cc:1918] T 483a9443000f4adbbab62a2a3710bcb2 P 46dcc2b09e12484e8e53004104b71230: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:57:22.671607 24311 log.cc:1196] T 483a9443000f4adbbab62a2a3710bcb2 P 46dcc2b09e12484e8e53004104b71230: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/ts-2/wal/wals/483a9443000f4adbbab62a2a3710bcb2
I20260806 02:57:22.672142 24311 ts_tablet_manager.cc:1939] T 483a9443000f4adbbab62a2a3710bcb2 P 46dcc2b09e12484e8e53004104b71230: Deleting consensus metadata
I20260806 02:57:22.672763 24314 tablet_replica.cc:330] T 483a9443000f4adbbab62a2a3710bcb2 P 97a954e4b78d444c95b9113ae889a585: stopping tablet replica
I20260806 02:57:22.673355 24314 raft_consensus.cc:2229] T 483a9443000f4adbbab62a2a3710bcb2 P 97a954e4b78d444c95b9113ae889a585 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 02:57:22.673947 24314 raft_consensus.cc:2258] T 483a9443000f4adbbab62a2a3710bcb2 P 97a954e4b78d444c95b9113ae889a585 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 02:57:22.674458 23565 catalog_manager.cc:4841] TS 46dcc2b09e12484e8e53004104b71230 (127.25.211.131:38803): tablet 483a9443000f4adbbab62a2a3710bcb2 (table test-table [id=a131d021eddc43ff9f578c5256f43d3a]) successfully deleted
I20260806 02:57:22.675694 24273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:22.656369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38728 (local address 127.25.211.129:42527)
0806 02:57:22.656600 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:22.656607 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:22.667052 (+ 10445us) server_negotiation.cc:407] Connection header received
0806 02:57:22.667362 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:22.667368 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:22.667481 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:22.667560 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:22.668338 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:22.669589 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:22.671638 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:22.672340 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:22.672448 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:22.673969 (+ 1521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:22.673982 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:22.674005 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:57:22.674822 (+ 817us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:22.674896 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:57:22.674923 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:22.675036 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:22.675200 (+ 164us) server_negotiation.cc:299] Negotiation successful
0806 02:57:22.675315 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260806 02:57:22.676385 24314 ts_tablet_manager.cc:1905] T 483a9443000f4adbbab62a2a3710bcb2 P 97a954e4b78d444c95b9113ae889a585: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 02:57:22.676749 23862 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:57:22.684307 24314 ts_tablet_manager.cc:1918] T 483a9443000f4adbbab62a2a3710bcb2 P 97a954e4b78d444c95b9113ae889a585: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 02:57:22.684540 24314 log.cc:1196] T 483a9443000f4adbbab62a2a3710bcb2 P 97a954e4b78d444c95b9113ae889a585: Deleting WAL directory at /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1785984707115702-26446-0/minicluster-data/ts-1/wal/wals/483a9443000f4adbbab62a2a3710bcb2
I20260806 02:57:22.685137 24314 ts_tablet_manager.cc:1939] T 483a9443000f4adbbab62a2a3710bcb2 P 97a954e4b78d444c95b9113ae889a585: Deleting consensus metadata
I20260806 02:57:22.687490 23564 catalog_manager.cc:4841] TS 97a954e4b78d444c95b9113ae889a585 (127.25.211.130:37771): tablet 483a9443000f4adbbab62a2a3710bcb2 (table test-table [id=a131d021eddc43ff9f578c5256f43d3a]) successfully deleted
I20260806 02:57:22.786191 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 23679
I20260806 02:57:22.810248 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 23872
I20260806 02:57:22.817349 24275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:22.811062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57772 (local address 127.25.211.130:37771)
0806 02:57:22.811266 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:22.811271 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:22.811291 (+ 20us) server_negotiation.cc:407] Connection header received
0806 02:57:22.811433 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:22.811437 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:22.811523 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:22.811611 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:22.812091 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:22.812879 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:22.814209 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:22.814723 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:22.814838 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:22.815941 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:22.815950 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:22.815967 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:57:22.816618 (+ 651us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:22.816672 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:57:22.816694 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:22.816782 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:22.816888 (+ 106us) server_negotiation.cc:299] Negotiation successful
0806 02:57:22.816989 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 02:57:22.818125 24057 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:57:22.906980 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 23872
I20260806 02:57:22.930001 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 24069
I20260806 02:57:22.936829 24271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:22.930806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43532 (local address 127.25.211.131:38803)
0806 02:57:22.931014 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:22.931019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:22.931041 (+ 22us) server_negotiation.cc:407] Connection header received
0806 02:57:22.931169 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:22.931173 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:22.931273 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:22.931354 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:22.931696 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:22.932423 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:22.933812 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:22.934248 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:22.934344 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:22.935444 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:22.935455 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:22.935472 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:57:22.936060 (+ 588us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:22.936115 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:57:22.936136 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:22.936231 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:22.936394 (+ 163us) server_negotiation.cc:299] Negotiation successful
0806 02:57:22.936489 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260806 02:57:22.937501 24245 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:57:23.026924 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 24069
I20260806 02:57:23.049883 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 23534
I20260806 02:57:23.051086 23629 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:57:23.161413 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 23534
2026-08-06T02:57:23Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3119 ms)
[----------] 6 tests from EncryptionPolicyTest (23968 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984707115702-26446-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:57:23 dist-test-slave-zj8q krb5kdc[24328](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:57:23 dist-test-slave-zj8q krb5kdc[24328](info): set up 2 sockets
Aug 06 02:57:23 dist-test-slave-zj8q krb5kdc[24328](info): commencing operation
krb5kdc: starting...
W20260806 02:57:24.402288 26446 mini_kdc.cc:121] Time spent starting KDC: real 1.192s user 0.002s sys 0.050s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Aug 06 02:57:24 dist-test-slave-zj8q krb5kdc[24328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:57:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:57:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984707115702-26446-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984707115702-26446-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:57:24.741364 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984707115702-26446-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984707115702-26446-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984707115702-26446-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43475
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984707115702-26446-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43475
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:57:25.095338 24356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:57:25.095650 24356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:57:25.095800 24356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:57:25.103009 24356 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:57:25.103123 24356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:57:25.103200 24356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:57:25.103284 24356 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:57:25.103345 24356 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:57:25.113482 24356 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984707115702-26446-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984707115702-26446-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43475
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984707115702-26446-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984707115702-26446-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:57:25.114923 24356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:57:25.116264 24356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:57:25.128525 24366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:25.128568 24361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:25.128464 24363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:57:25.129623 24356 server_base.cc:1034] running on GCE node
I20260806 02:57:25.130750 24356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:57:25.132318 24356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:57:25.133533 24356 hybrid_clock.cc:648] HybridClock initialized: now 1785985045133490 us; error 30 us; skew 500 ppm
I20260806 02:57:25.133965 24356 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:57:25 dist-test-slave-zj8q krb5kdc[24328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985045, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:57:25.137391 24356 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:57:25.139092 24356 webserver.cc:460] Webserver started at http://127.0.0.1:40231/ using document root <none> and password file <none>
I20260806 02:57:25.139822 24356 fs_manager.cc:359] Metadata directory not provided
I20260806 02:57:25.139946 24356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:57:25.140218 24356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:57:25.143473 24356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984707115702-26446-0/minicluster-data/master-0/data/instance:
uuid: "2649c092d1474287aab1e4524f2b194b"
format_stamp: "Formatted at 2026-08-06 02:57:25 on dist-test-slave-zj8q"
server_key: "b49b4a66c3eba072f930ce94aa71e48e"
server_key_iv: "7bf28cf1e5c5af69e2909fc7da503eb8"
server_key_version: "encryptionkey@0"
I20260806 02:57:25.144286 24356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984707115702-26446-0/minicluster-data/master-0/wal/instance:
uuid: "2649c092d1474287aab1e4524f2b194b"
format_stamp: "Formatted at 2026-08-06 02:57:25 on dist-test-slave-zj8q"
server_key: "b49b4a66c3eba072f930ce94aa71e48e"
server_key_iv: "7bf28cf1e5c5af69e2909fc7da503eb8"
server_key_version: "encryptionkey@0"
I20260806 02:57:25.150081 24356 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260806 02:57:25.154058 24376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:25.155675 24356 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:57:25.155874 24356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984707115702-26446-0/minicluster-data/master-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984707115702-26446-0/minicluster-data/master-0/wal
uuid: "2649c092d1474287aab1e4524f2b194b"
format_stamp: "Formatted at 2026-08-06 02:57:25 on dist-test-slave-zj8q"
server_key: "b49b4a66c3eba072f930ce94aa71e48e"
server_key_iv: "7bf28cf1e5c5af69e2909fc7da503eb8"
server_key_version: "encryptionkey@0"
I20260806 02:57:25.156071 24356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984707115702-26446-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984707115702-26446-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984707115702-26446-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:57:25.181430 24356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:57:25.186127 24356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:57:25.186432 24356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:57:25.206066 24356 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43475
I20260806 02:57:25.206099 24446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43475 every 8 connection(s)
I20260806 02:57:25.207654 24356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
I20260806 02:57:25.212085 24447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:57:25.216382 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 24356
I20260806 02:57:25.216614 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1785984707115702-26446-0/minicluster-data/master-0/wal/instance
I20260806 02:57:25.217324 26446 external_mini_cluster.cc:1428] Setting key 9eb1604ce9c18a58d31ae4be805bcea4
I20260806 02:57:25.224387 24447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2649c092d1474287aab1e4524f2b194b: Bootstrap starting.
I20260806 02:57:25.228564 24447 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2649c092d1474287aab1e4524f2b194b: Neither blocks nor log segments found. Creating new log.
I20260806 02:57:25.229872 24447 log.cc:824] T 00000000000000000000000000000000 P 2649c092d1474287aab1e4524f2b194b: Log is configured to *not* fsync() on all Append() calls
I20260806 02:57:25.233884 24447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2649c092d1474287aab1e4524f2b194b: No bootstrap required, opened a new log
Aug 06 02:57:25 dist-test-slave-zj8q krb5kdc[24328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 02:57:25.243623 24447 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2649c092d1474287aab1e4524f2b194b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2649c092d1474287aab1e4524f2b194b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43475 } }
I20260806 02:57:25.244042 24450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:25.219501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44785 (local address 127.0.0.1:43475)
0806 02:57:25.220366 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:25.220378 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:25.220424 (+ 46us) server_negotiation.cc:407] Connection header received
0806 02:57:25.222632 (+ 2208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:25.222662 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:25.223116 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:25.223465 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:25.225488 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:25.227043 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:25.229003 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:25.229851 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:25.230023 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:25.235570 (+ 5547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:25.235579 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:25.235597 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:57:25.235639 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:25.239066 (+ 3427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:25.240222 (+ 1156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:25.240230 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:25.240241 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:25.240454 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:25.241019 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:25.241023 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:25.241025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:25.241700 (+ 675us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:25.241954 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:25.242535 (+ 581us) server_negotiation.cc:299] Negotiation successful
0806 02:57:25.242904 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":402,"threads_started":1}
I20260806 02:57:25.244117 24447 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2649c092d1474287aab1e4524f2b194b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:57:25.244237 24447 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2649c092d1474287aab1e4524f2b194b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2649c092d1474287aab1e4524f2b194b, State: Initialized, Role: FOLLOWER
I20260806 02:57:25.244953 24447 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2649c092d1474287aab1e4524f2b194b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2649c092d1474287aab1e4524f2b194b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43475 } }
I20260806 02:57:25.245303 24447 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2649c092d1474287aab1e4524f2b194b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:57:25.245450 24447 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2649c092d1474287aab1e4524f2b194b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:57:25.245697 24447 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2649c092d1474287aab1e4524f2b194b [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:57:25.248548 24447 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2649c092d1474287aab1e4524f2b194b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2649c092d1474287aab1e4524f2b194b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43475 } }
I20260806 02:57:25.248857 26446 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:57:25.249125 24447 leader_election.cc:302] T 00000000000000000000000000000000 P 2649c092d1474287aab1e4524f2b194b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2649c092d1474287aab1e4524f2b194b; no voters:
I20260806 02:57:25.249758 24447 leader_election.cc:288] T 00000000000000000000000000000000 P 2649c092d1474287aab1e4524f2b194b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:57:25.249919 24457 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2649c092d1474287aab1e4524f2b194b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:57:25.251809 24447 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2649c092d1474287aab1e4524f2b194b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:57:25.257411 24457 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2649c092d1474287aab1e4524f2b194b [term 1 LEADER]: Becoming Leader. State: Replica: 2649c092d1474287aab1e4524f2b194b, State: Running, Role: LEADER
I20260806 02:57:25.258195 24457 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2649c092d1474287aab1e4524f2b194b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2649c092d1474287aab1e4524f2b194b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43475 } }
I20260806 02:57:25.261009 24458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2649c092d1474287aab1e4524f2b194b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2649c092d1474287aab1e4524f2b194b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2649c092d1474287aab1e4524f2b194b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43475 } } }
I20260806 02:57:25.261420 24458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2649c092d1474287aab1e4524f2b194b [sys.catalog]: This master's current role is: LEADER
I20260806 02:57:25.262408 24489 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:57:25.262248 24459 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2649c092d1474287aab1e4524f2b194b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2649c092d1474287aab1e4524f2b194b. Latest consensus state: current_term: 1 leader_uuid: "2649c092d1474287aab1e4524f2b194b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2649c092d1474287aab1e4524f2b194b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43475 } } }
I20260806 02:57:25.262609 24459 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2649c092d1474287aab1e4524f2b194b [sys.catalog]: This master's current role is: LEADER
I20260806 02:57:25.269403 24489 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:57:25.278463 24450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:25.259423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47309 (local address 127.0.0.1:43475)
0806 02:57:25.259630 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:25.259637 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:25.260005 (+ 368us) server_negotiation.cc:407] Connection header received
0806 02:57:25.260224 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:25.260230 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:25.260324 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:25.260437 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:25.263428 (+ 2991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:25.264696 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:25.266497 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:25.267213 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:25.267315 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:25.270903 (+ 3588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:25.270909 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:25.270914 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:57:25.270960 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:25.274832 (+ 3872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:25.275916 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:25.275922 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:25.275924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:25.276080 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:25.276652 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:25.276657 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:25.276660 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:25.277187 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:25.277435 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:25.277885 (+ 450us) server_negotiation.cc:299] Negotiation successful
0806 02:57:25.278079 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 02:57:25.279484 24489 catalog_manager.cc:1347] Generated new cluster ID: 8400d9dee7224ec1a2284848dd3b9c3e
I20260806 02:57:25.279635 24489 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:57:25.288538 24489 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:57:25.290115 24489 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:57:25.300696 24489 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2649c092d1474287aab1e4524f2b194b: Generated new TSK 0
I20260806 02:57:25.301744 24489 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:57:25.310395 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 24356
I20260806 02:57:25.312577 24440 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:57:25.404181 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 24356
W20260806 02:57:25.428462 24324 connection.cc:535] client connection to 127.0.0.1:43475 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43475): Connection reset by peer (error 104)
2026-08-06T02:57:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2238 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984707115702-26446-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:57:25 dist-test-slave-zj8q krb5kdc[24500](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:57:25 dist-test-slave-zj8q krb5kdc[24500](info): set up 2 sockets
Aug 06 02:57:25 dist-test-slave-zj8q krb5kdc[24500](info): commencing operation
krb5kdc: starting...
W20260806 02:57:26.640547 26446 mini_kdc.cc:121] Time spent starting KDC: real 1.192s user 0.004s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Aug 06 02:57:26 dist-test-slave-zj8q krb5kdc[24500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:57:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:57:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984707115702-26446-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984707115702-26446-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:57:26.973596 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984707115702-26446-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984707115702-26446-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984707115702-26446-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36799
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984707115702-26446-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36799
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:57:27.325985 24522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:57:27.326370 24522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:57:27.326514 24522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:57:27.334123 24522 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:57:27.334308 24522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:57:27.334388 24522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:57:27.334477 24522 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:57:27.334534 24522 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:57:27.345014 24522 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984707115702-26446-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984707115702-26446-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36799
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984707115702-26446-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984707115702-26446-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:57:27.346580 24522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:57:27.348021 24522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:57:27.360162 24530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:27.360193 24529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:27.360710 24533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:57:27.361245 24522 server_base.cc:1034] running on GCE node
I20260806 02:57:27.362072 24522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:57:27.363602 24522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:57:27.364820 24522 hybrid_clock.cc:648] HybridClock initialized: now 1785985047364784 us; error 52 us; skew 500 ppm
I20260806 02:57:27.365164 24522 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:57:27 dist-test-slave-zj8q krb5kdc[24500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985047, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:57:27.368292 24522 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:57:27.369836 24522 webserver.cc:460] Webserver started at http://127.0.0.1:46319/ using document root <none> and password file <none>
I20260806 02:57:27.370589 24522 fs_manager.cc:359] Metadata directory not provided
I20260806 02:57:27.370718 24522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:57:27.371025 24522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:57:27.374188 24522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984707115702-26446-0/minicluster-data/master-0/data/instance:
uuid: "d8bcca97400642c9ab9c88258ee2871a"
format_stamp: "Formatted at 2026-08-06 02:57:27 on dist-test-slave-zj8q"
server_key: "44f51ef2cdb8e73fdc643eefeec5700b"
server_key_iv: "0e2736c7273aa8dbc5778a3cd396bb38"
server_key_version: "encryptionkey@0"
I20260806 02:57:27.375026 24522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984707115702-26446-0/minicluster-data/master-0/wal/instance:
uuid: "d8bcca97400642c9ab9c88258ee2871a"
format_stamp: "Formatted at 2026-08-06 02:57:27 on dist-test-slave-zj8q"
server_key: "44f51ef2cdb8e73fdc643eefeec5700b"
server_key_iv: "0e2736c7273aa8dbc5778a3cd396bb38"
server_key_version: "encryptionkey@0"
I20260806 02:57:27.380623 24522 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260806 02:57:27.384735 24546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:27.386264 24522 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:57:27.386464 24522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984707115702-26446-0/minicluster-data/master-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984707115702-26446-0/minicluster-data/master-0/wal
uuid: "d8bcca97400642c9ab9c88258ee2871a"
format_stamp: "Formatted at 2026-08-06 02:57:27 on dist-test-slave-zj8q"
server_key: "44f51ef2cdb8e73fdc643eefeec5700b"
server_key_iv: "0e2736c7273aa8dbc5778a3cd396bb38"
server_key_version: "encryptionkey@0"
I20260806 02:57:27.386716 24522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984707115702-26446-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984707115702-26446-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984707115702-26446-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:57:27.402037 24522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:57:27.406522 24522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:57:27.406814 24522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:57:27.425472 24522 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36799
I20260806 02:57:27.425501 24616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36799 every 8 connection(s)
I20260806 02:57:27.426942 24522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
I20260806 02:57:27.430701 24619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:57:27.437165 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 24522
I20260806 02:57:27.437336 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1785984707115702-26446-0/minicluster-data/master-0/wal/instance
I20260806 02:57:27.437798 26446 external_mini_cluster.cc:1428] Setting key 6edf34d8e792cd15f64e14c5c4ef5a21
I20260806 02:57:27.442373 24619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8bcca97400642c9ab9c88258ee2871a: Bootstrap starting.
I20260806 02:57:27.446005 24619 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8bcca97400642c9ab9c88258ee2871a: Neither blocks nor log segments found. Creating new log.
I20260806 02:57:27.447412 24619 log.cc:824] T 00000000000000000000000000000000 P d8bcca97400642c9ab9c88258ee2871a: Log is configured to *not* fsync() on all Append() calls
I20260806 02:57:27.450831 24619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8bcca97400642c9ab9c88258ee2871a: No bootstrap required, opened a new log
I20260806 02:57:27.452083 24625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:27.439706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33939 (local address 127.0.0.1:36799)
0806 02:57:27.440589 (+ 883us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:27.440602 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:27.440647 (+ 45us) server_negotiation.cc:407] Connection header received
0806 02:57:27.442506 (+ 1859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:27.442539 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:27.442984 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:27.443329 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:27.443970 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:27.445342 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:27.447296 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:27.448327 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:27.448533 (+ 206us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:27.449695 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:27.449703 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:27.449720 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:57:27.450329 (+ 609us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:27.450405 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:57:27.450435 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:27.450564 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:27.450831 (+ 267us) server_negotiation.cc:299] Negotiation successful
0806 02:57:27.451001 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":418,"threads_started":1}
I20260806 02:57:27.455967 26446 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:57:27.460574 24619 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8bcca97400642c9ab9c88258ee2871a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8bcca97400642c9ab9c88258ee2871a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36799 } }
I20260806 02:57:27.461153 24619 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8bcca97400642c9ab9c88258ee2871a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:57:27.461261 24619 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8bcca97400642c9ab9c88258ee2871a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8bcca97400642c9ab9c88258ee2871a, State: Initialized, Role: FOLLOWER
I20260806 02:57:27.462044 24619 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8bcca97400642c9ab9c88258ee2871a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8bcca97400642c9ab9c88258ee2871a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36799 } }
I20260806 02:57:27.462392 24619 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8bcca97400642c9ab9c88258ee2871a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:57:27.462554 24619 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8bcca97400642c9ab9c88258ee2871a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:57:27.462786 24619 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8bcca97400642c9ab9c88258ee2871a [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:57:27.465883 24619 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8bcca97400642c9ab9c88258ee2871a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8bcca97400642c9ab9c88258ee2871a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36799 } }
I20260806 02:57:27.466599 24619 leader_election.cc:302] T 00000000000000000000000000000000 P d8bcca97400642c9ab9c88258ee2871a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8bcca97400642c9ab9c88258ee2871a; no voters:
I20260806 02:57:27.467303 24619 leader_election.cc:288] T 00000000000000000000000000000000 P d8bcca97400642c9ab9c88258ee2871a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:57:27.467393 24634 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8bcca97400642c9ab9c88258ee2871a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:57:27.468101 24634 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8bcca97400642c9ab9c88258ee2871a [term 1 LEADER]: Becoming Leader. State: Replica: d8bcca97400642c9ab9c88258ee2871a, State: Running, Role: LEADER
I20260806 02:57:27.468870 24634 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8bcca97400642c9ab9c88258ee2871a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8bcca97400642c9ab9c88258ee2871a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36799 } }
I20260806 02:57:27.469537 24625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:27.461722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59937 (local address 127.0.0.1:36799)
0806 02:57:27.461917 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:27.461925 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:27.462457 (+ 532us) server_negotiation.cc:407] Connection header received
0806 02:57:27.462698 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:27.462704 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:27.462789 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:27.462887 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:27.463556 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:27.464429 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:27.466273 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:27.466775 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:27.466885 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:27.468198 (+ 1313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:27.468205 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:27.468213 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:57:27.468237 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:27.468314 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:57:27.468341 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:27.468470 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:27.469170 (+ 700us) server_negotiation.cc:299] Negotiation successful
0806 02:57:27.469228 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 02:57:27.470022 24619 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8bcca97400642c9ab9c88258ee2871a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:57:27.471776 24635 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8bcca97400642c9ab9c88258ee2871a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8bcca97400642c9ab9c88258ee2871a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8bcca97400642c9ab9c88258ee2871a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36799 } } }
I20260806 02:57:27.472092 24635 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8bcca97400642c9ab9c88258ee2871a [sys.catalog]: This master's current role is: LEADER
I20260806 02:57:27.472994 24639 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:57:27.472780 24636 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8bcca97400642c9ab9c88258ee2871a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8bcca97400642c9ab9c88258ee2871a. Latest consensus state: current_term: 1 leader_uuid: "d8bcca97400642c9ab9c88258ee2871a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8bcca97400642c9ab9c88258ee2871a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36799 } } }
I20260806 02:57:27.473151 24636 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8bcca97400642c9ab9c88258ee2871a [sys.catalog]: This master's current role is: LEADER
I20260806 02:57:27.479712 24639 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:57:27.490326 24639 catalog_manager.cc:1347] Generated new cluster ID: a5d84fdb1b11405f8ce51b56cf6f5cf5
I20260806 02:57:27.490684 24639 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:57:27.502321 24639 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:57:27.503717 24639 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:57:27.514763 24639 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8bcca97400642c9ab9c88258ee2871a: Generated new TSK 0
I20260806 02:57:27.515650 24639 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:57:27.521420 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 24522
W20260806 02:57:27.522552 24609 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33939
W20260806 02:57:27.523306 26446 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 24522: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 02:57:27.523471 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 24522
W20260806 02:57:27.547890 24498 connection.cc:535] client connection to 127.0.0.1:36799 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36799): Connection reset by peer (error 104)
2026-08-06T02:57:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2119 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984707115702-26446-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:57:27 dist-test-slave-zj8q krb5kdc[24663](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:57:27 dist-test-slave-zj8q krb5kdc[24663](info): set up 2 sockets
Aug 06 02:57:27 dist-test-slave-zj8q krb5kdc[24663](info): commencing operation
krb5kdc: starting...
W20260806 02:57:28.759094 26446 mini_kdc.cc:121] Time spent starting KDC: real 1.192s user 0.001s sys 0.051s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Aug 06 02:57:28 dist-test-slave-zj8q krb5kdc[24663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:57:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:57:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984707115702-26446-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984707115702-26446-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:57:29.095676 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984707115702-26446-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984707115702-26446-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984707115702-26446-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36857
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984707115702-26446-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36857
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:57:29.450222 24684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:57:29.450562 24684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:57:29.450702 24684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:57:29.458215 24684 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:57:29.458351 24684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:57:29.458431 24684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:57:29.458514 24684 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:57:29.458635 24684 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:57:29.469250 24684 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984707115702-26446-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984707115702-26446-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36857
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984707115702-26446-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984707115702-26446-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:57:29.470634 24684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:57:29.471938 24684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:57:29.483739 24690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:29.483709 24693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:29.483708 24691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:57:29.485239 24684 server_base.cc:1034] running on GCE node
I20260806 02:57:29.486126 24684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:57:29.487718 24684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:57:29.488936 24684 hybrid_clock.cc:648] HybridClock initialized: now 1785985049488885 us; error 45 us; skew 500 ppm
I20260806 02:57:29.489292 24684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:57:29 dist-test-slave-zj8q krb5kdc[24663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985049, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:57:29.495796 24684 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:57:29.499343 24684 webserver.cc:460] Webserver started at http://127.0.0.1:45175/ using document root <none> and password file <none>
I20260806 02:57:29.500090 24684 fs_manager.cc:359] Metadata directory not provided
I20260806 02:57:29.500214 24684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:57:29.500536 24684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:57:29.503700 24684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984707115702-26446-0/minicluster-data/master-0/data/instance:
uuid: "4c2cce8c4aaa4a939f0401cad6c39a46"
format_stamp: "Formatted at 2026-08-06 02:57:29 on dist-test-slave-zj8q"
server_key: "8efe7b24d735fc1dd8efc1fa11a2313a"
server_key_iv: "f1d145cc8fc602f8741b9a53c89db46d"
server_key_version: "encryptionkey@0"
I20260806 02:57:29.504455 24684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984707115702-26446-0/minicluster-data/master-0/wal/instance:
uuid: "4c2cce8c4aaa4a939f0401cad6c39a46"
format_stamp: "Formatted at 2026-08-06 02:57:29 on dist-test-slave-zj8q"
server_key: "8efe7b24d735fc1dd8efc1fa11a2313a"
server_key_iv: "f1d145cc8fc602f8741b9a53c89db46d"
server_key_version: "encryptionkey@0"
I20260806 02:57:29.509825 24684 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260806 02:57:29.513518 24703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:29.515131 24684 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:57:29.515316 24684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984707115702-26446-0/minicluster-data/master-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984707115702-26446-0/minicluster-data/master-0/wal
uuid: "4c2cce8c4aaa4a939f0401cad6c39a46"
format_stamp: "Formatted at 2026-08-06 02:57:29 on dist-test-slave-zj8q"
server_key: "8efe7b24d735fc1dd8efc1fa11a2313a"
server_key_iv: "f1d145cc8fc602f8741b9a53c89db46d"
server_key_version: "encryptionkey@0"
I20260806 02:57:29.515516 24684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984707115702-26446-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984707115702-26446-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984707115702-26446-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:57:29.521458 24684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:57:29.525633 24684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:57:29.525933 24684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:57:29.543447 24684 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36857
I20260806 02:57:29.543485 24782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36857 every 8 connection(s)
I20260806 02:57:29.544962 24684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
I20260806 02:57:29.549216 24783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:57:29.550242 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 24684
I20260806 02:57:29.550414 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1785984707115702-26446-0/minicluster-data/master-0/wal/instance
I20260806 02:57:29.550873 26446 external_mini_cluster.cc:1428] Setting key a4d4510efd1fd637f2c5ebd03b881b10
I20260806 02:57:29.559900 24789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:29.552831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35981 (local address 127.0.0.1:36857)
0806 02:57:29.553701 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:29.553718 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:29.553769 (+ 51us) server_negotiation.cc:407] Connection header received
0806 02:57:29.555978 (+ 2209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:29.556013 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:29.556505 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:29.556983 (+ 478us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:29.557927 (+ 944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:29.557934 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:29.557951 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:57:29.557967 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:29.558036 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:57:29.558067 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:29.558325 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:29.558448 (+ 123us) server_negotiation.cc:299] Negotiation successful
0806 02:57:29.558646 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":348,"threads_started":1}
I20260806 02:57:29.564142 26446 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:57:29.564435 24783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c2cce8c4aaa4a939f0401cad6c39a46: Bootstrap starting.
I20260806 02:57:29.567910 24783 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c2cce8c4aaa4a939f0401cad6c39a46: Neither blocks nor log segments found. Creating new log.
I20260806 02:57:29.569199 24783 log.cc:824] T 00000000000000000000000000000000 P 4c2cce8c4aaa4a939f0401cad6c39a46: Log is configured to *not* fsync() on all Append() calls
I20260806 02:57:29.571611 24789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:29.568811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38355 (local address 127.0.0.1:36857)
0806 02:57:29.569017 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:29.569022 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:29.569330 (+ 308us) server_negotiation.cc:407] Connection header received
0806 02:57:29.569512 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:29.569517 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:29.569581 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:29.569710 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:29.570898 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:29.570904 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:29.570909 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:57:29.570928 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:29.570974 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:57:29.570995 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:29.571153 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:29.571269 (+ 116us) server_negotiation.cc:299] Negotiation successful
0806 02:57:29.571331 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260806 02:57:29.573369 24783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c2cce8c4aaa4a939f0401cad6c39a46: No bootstrap required, opened a new log
I20260806 02:57:29.581142 24783 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c2cce8c4aaa4a939f0401cad6c39a46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c2cce8c4aaa4a939f0401cad6c39a46" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36857 } }
I20260806 02:57:29.581597 24783 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c2cce8c4aaa4a939f0401cad6c39a46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:57:29.581698 24783 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c2cce8c4aaa4a939f0401cad6c39a46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c2cce8c4aaa4a939f0401cad6c39a46, State: Initialized, Role: FOLLOWER
I20260806 02:57:29.582463 24783 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c2cce8c4aaa4a939f0401cad6c39a46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c2cce8c4aaa4a939f0401cad6c39a46" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36857 } }
I20260806 02:57:29.582739 24783 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c2cce8c4aaa4a939f0401cad6c39a46 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:57:29.582890 24783 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c2cce8c4aaa4a939f0401cad6c39a46 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:57:29.583067 24783 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c2cce8c4aaa4a939f0401cad6c39a46 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:57:29.585561 24783 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c2cce8c4aaa4a939f0401cad6c39a46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c2cce8c4aaa4a939f0401cad6c39a46" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36857 } }
I20260806 02:57:29.586117 24783 leader_election.cc:302] T 00000000000000000000000000000000 P 4c2cce8c4aaa4a939f0401cad6c39a46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c2cce8c4aaa4a939f0401cad6c39a46; no voters:
I20260806 02:57:29.586791 24783 leader_election.cc:288] T 00000000000000000000000000000000 P 4c2cce8c4aaa4a939f0401cad6c39a46 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:57:29.586943 24805 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c2cce8c4aaa4a939f0401cad6c39a46 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:57:29.587590 24805 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c2cce8c4aaa4a939f0401cad6c39a46 [term 1 LEADER]: Becoming Leader. State: Replica: 4c2cce8c4aaa4a939f0401cad6c39a46, State: Running, Role: LEADER
I20260806 02:57:29.588195 24805 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c2cce8c4aaa4a939f0401cad6c39a46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c2cce8c4aaa4a939f0401cad6c39a46" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36857 } }
I20260806 02:57:29.588927 24783 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c2cce8c4aaa4a939f0401cad6c39a46 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:57:29.591055 24807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c2cce8c4aaa4a939f0401cad6c39a46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c2cce8c4aaa4a939f0401cad6c39a46. Latest consensus state: current_term: 1 leader_uuid: "4c2cce8c4aaa4a939f0401cad6c39a46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c2cce8c4aaa4a939f0401cad6c39a46" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36857 } } }
I20260806 02:57:29.591524 24807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c2cce8c4aaa4a939f0401cad6c39a46 [sys.catalog]: This master's current role is: LEADER
I20260806 02:57:29.592314 24806 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c2cce8c4aaa4a939f0401cad6c39a46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c2cce8c4aaa4a939f0401cad6c39a46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c2cce8c4aaa4a939f0401cad6c39a46" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36857 } } }
I20260806 02:57:29.593187 24806 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c2cce8c4aaa4a939f0401cad6c39a46 [sys.catalog]: This master's current role is: LEADER
I20260806 02:57:29.595602 24818 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:57:29.600780 24818 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:57:29.608882 24818 catalog_manager.cc:1347] Generated new cluster ID: 3af9092f273f4f7e9ff293831c991e52
I20260806 02:57:29.609050 24818 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:57:29.633078 24818 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:57:29.633610 24818 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:57:29.643766 24818 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c2cce8c4aaa4a939f0401cad6c39a46: Generated new TSK 0
I20260806 02:57:29.644702 24818 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:57:29.660964 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 24684
W20260806 02:57:29.662251 24777 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:35981
W20260806 02:57:29.663143 26446 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 24684: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 02:57:29.663295 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 24684
2026-08-06T02:57:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2142 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984707115702-26446-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:57:29 dist-test-slave-zj8q krb5kdc[24835](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:57:29 dist-test-slave-zj8q krb5kdc[24835](info): set up 2 sockets
Aug 06 02:57:29 dist-test-slave-zj8q krb5kdc[24835](info): commencing operation
krb5kdc: starting...
W20260806 02:57:30.903412 26446 mini_kdc.cc:121] Time spent starting KDC: real 1.193s user 0.000s sys 0.051s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Aug 06 02:57:31 dist-test-slave-zj8q krb5kdc[24835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:57:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:57:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984707115702-26446-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984707115702-26446-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:57:31.251632 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984707115702-26446-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984707115702-26446-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984707115702-26446-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42101
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984707115702-26446-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:57:31.605011 24859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:57:31.605350 24859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:57:31.605475 24859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:57:31.613143 24859 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:57:31.613286 24859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:57:31.613339 24859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:57:31.613430 24859 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:57:31.613482 24859 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:57:31.623811 24859 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984707115702-26446-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984707115702-26446-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984707115702-26446-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984707115702-26446-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:57:31.625293 24859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:57:31.626724 24859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:57:31.638681 24867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:31.638727 24873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:31.638758 24866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:57:31.639895 24859 server_base.cc:1034] running on GCE node
I20260806 02:57:31.640815 24859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:57:31.642395 24859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:57:31.643663 24859 hybrid_clock.cc:648] HybridClock initialized: now 1785985051643607 us; error 59 us; skew 500 ppm
I20260806 02:57:31.644038 24859 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:57:31 dist-test-slave-zj8q krb5kdc[24835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985051, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:57:31.647599 24859 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:57:31.649353 24859 webserver.cc:460] Webserver started at http://127.0.0.1:46835/ using document root <none> and password file <none>
I20260806 02:57:31.650110 24859 fs_manager.cc:359] Metadata directory not provided
I20260806 02:57:31.650296 24859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:57:31.650643 24859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:57:31.654053 24859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984707115702-26446-0/minicluster-data/master-0/data/instance:
uuid: "b0e8eb41b160423892fea9a52de5ee42"
format_stamp: "Formatted at 2026-08-06 02:57:31 on dist-test-slave-zj8q"
server_key: "b6bea82ed2b7dcd8a2e4ad100670b09c"
server_key_iv: "3cb154ef0f2565b0a75725228b98b1d6"
server_key_version: "encryptionkey@0"
I20260806 02:57:31.655025 24859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984707115702-26446-0/minicluster-data/master-0/wal/instance:
uuid: "b0e8eb41b160423892fea9a52de5ee42"
format_stamp: "Formatted at 2026-08-06 02:57:31 on dist-test-slave-zj8q"
server_key: "b6bea82ed2b7dcd8a2e4ad100670b09c"
server_key_iv: "3cb154ef0f2565b0a75725228b98b1d6"
server_key_version: "encryptionkey@0"
I20260806 02:57:31.661258 24859 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260806 02:57:31.665381 24882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:31.667105 24859 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:57:31.667500 24859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984707115702-26446-0/minicluster-data/master-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984707115702-26446-0/minicluster-data/master-0/wal
uuid: "b0e8eb41b160423892fea9a52de5ee42"
format_stamp: "Formatted at 2026-08-06 02:57:31 on dist-test-slave-zj8q"
server_key: "b6bea82ed2b7dcd8a2e4ad100670b09c"
server_key_iv: "3cb154ef0f2565b0a75725228b98b1d6"
server_key_version: "encryptionkey@0"
I20260806 02:57:31.667747 24859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984707115702-26446-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984707115702-26446-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984707115702-26446-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:57:31.686375 24859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:57:31.691270 24859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:57:31.691591 24859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:57:31.710758 24859 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42101
I20260806 02:57:31.710788 24955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42101 every 8 connection(s)
I20260806 02:57:31.712169 24859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
I20260806 02:57:31.716102 24956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:57:31.717082 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 24859
I20260806 02:57:31.717268 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1785984707115702-26446-0/minicluster-data/master-0/wal/instance
I20260806 02:57:31.717850 26446 external_mini_cluster.cc:1428] Setting key 9c948204f89df6f288ce873a2c5a9ab6
I20260806 02:57:31.731031 24956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0e8eb41b160423892fea9a52de5ee42: Bootstrap starting.
Aug 06 02:57:31 dist-test-slave-zj8q krb5kdc[24835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 02:57:31.735193 24956 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0e8eb41b160423892fea9a52de5ee42: Neither blocks nor log segments found. Creating new log.
I20260806 02:57:31.736330 24956 log.cc:824] T 00000000000000000000000000000000 P b0e8eb41b160423892fea9a52de5ee42: Log is configured to *not* fsync() on all Append() calls
I20260806 02:57:31.739890 24956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0e8eb41b160423892fea9a52de5ee42: No bootstrap required, opened a new log
I20260806 02:57:31.742704 24960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:31.719805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50929 (local address 127.0.0.1:42101)
0806 02:57:31.720581 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:31.720593 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:31.720648 (+ 55us) server_negotiation.cc:407] Connection header received
0806 02:57:31.722603 (+ 1955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:31.722631 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:31.723060 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:31.723394 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:31.725359 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:31.726819 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:31.728723 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:31.729485 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:31.729633 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:31.734784 (+ 5151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:31.734792 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:31.734808 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:57:31.734850 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:31.738217 (+ 3367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:31.739143 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:31.739150 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:31.739156 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:31.739329 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:31.739772 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:31.739775 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:31.739778 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:31.740386 (+ 608us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:31.740630 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:31.741146 (+ 516us) server_negotiation.cc:299] Negotiation successful
0806 02:57:31.741514 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":337,"threads_started":1}
I20260806 02:57:31.746573 26446 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:57:31.749339 24956 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0e8eb41b160423892fea9a52de5ee42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e8eb41b160423892fea9a52de5ee42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42101 } }
I20260806 02:57:31.749904 24956 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0e8eb41b160423892fea9a52de5ee42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:57:31.750031 24956 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0e8eb41b160423892fea9a52de5ee42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0e8eb41b160423892fea9a52de5ee42, State: Initialized, Role: FOLLOWER
I20260806 02:57:31.750833 24956 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0e8eb41b160423892fea9a52de5ee42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e8eb41b160423892fea9a52de5ee42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42101 } }
I20260806 02:57:31.751169 24956 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0e8eb41b160423892fea9a52de5ee42 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:57:31.751331 24956 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0e8eb41b160423892fea9a52de5ee42 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:57:31.751523 24956 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0e8eb41b160423892fea9a52de5ee42 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:57:31.754251 24956 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0e8eb41b160423892fea9a52de5ee42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e8eb41b160423892fea9a52de5ee42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42101 } }
I20260806 02:57:31.754878 24956 leader_election.cc:302] T 00000000000000000000000000000000 P b0e8eb41b160423892fea9a52de5ee42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0e8eb41b160423892fea9a52de5ee42; no voters:
I20260806 02:57:31.755605 24956 leader_election.cc:288] T 00000000000000000000000000000000 P b0e8eb41b160423892fea9a52de5ee42 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:57:31.755716 24973 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0e8eb41b160423892fea9a52de5ee42 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:57:31.756896 24973 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0e8eb41b160423892fea9a52de5ee42 [term 1 LEADER]: Becoming Leader. State: Replica: b0e8eb41b160423892fea9a52de5ee42, State: Running, Role: LEADER
I20260806 02:57:31.757572 24973 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0e8eb41b160423892fea9a52de5ee42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e8eb41b160423892fea9a52de5ee42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42101 } }
I20260806 02:57:31.757939 24956 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0e8eb41b160423892fea9a52de5ee42 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:57:31.764061 24974 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0e8eb41b160423892fea9a52de5ee42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0e8eb41b160423892fea9a52de5ee42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e8eb41b160423892fea9a52de5ee42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42101 } } }
I20260806 02:57:31.764496 24974 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0e8eb41b160423892fea9a52de5ee42 [sys.catalog]: This master's current role is: LEADER
I20260806 02:57:31.764990 24975 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0e8eb41b160423892fea9a52de5ee42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0e8eb41b160423892fea9a52de5ee42. Latest consensus state: current_term: 1 leader_uuid: "b0e8eb41b160423892fea9a52de5ee42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0e8eb41b160423892fea9a52de5ee42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42101 } } }
I20260806 02:57:31.765354 24975 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0e8eb41b160423892fea9a52de5ee42 [sys.catalog]: This master's current role is: LEADER
I20260806 02:57:31.767611 24986 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:57:31.770371 24960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:31.752126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53315 (local address 127.0.0.1:42101)
0806 02:57:31.752334 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:31.752341 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:31.752668 (+ 327us) server_negotiation.cc:407] Connection header received
0806 02:57:31.752830 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:31.752835 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:31.752919 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:31.753038 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:31.754613 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:31.755719 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:31.757596 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:31.758286 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:31.758417 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:31.762031 (+ 3614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:31.762041 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:31.762047 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:57:31.762096 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:31.766818 (+ 4722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:31.767802 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:31.767812 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:31.767816 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:31.767977 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:31.768445 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:31.768451 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:31.768454 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:31.768930 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:31.769235 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:31.769596 (+ 361us) server_negotiation.cc:299] Negotiation successful
0806 02:57:31.769861 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 02:57:31.775913 24986 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:57:31.786284 24986 catalog_manager.cc:1347] Generated new cluster ID: 61d857ba5e174f76af0d4c7263755372
I20260806 02:57:31.786432 24986 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:57:31.805855 24986 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:57:31.807937 24986 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:57:31.818988 24986 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0e8eb41b160423892fea9a52de5ee42: Generated new TSK 0
I20260806 02:57:31.820190 24986 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:57:31.831636 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 24859
I20260806 02:57:31.834015 24949 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:57:31.927734 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 24859
2026-08-06T02:57:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2259 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984707115702-26446-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:57:32 dist-test-slave-zj8q krb5kdc[25008](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:57:32 dist-test-slave-zj8q krb5kdc[25008](info): set up 2 sockets
Aug 06 02:57:32 dist-test-slave-zj8q krb5kdc[25008](info): commencing operation
krb5kdc: starting...
W20260806 02:57:33.157171 26446 mini_kdc.cc:121] Time spent starting KDC: real 1.188s user 0.003s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Aug 06 02:57:33 dist-test-slave-zj8q krb5kdc[25008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:57:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:57:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984707115702-26446-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984707115702-26446-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:57:33.497310 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984707115702-26446-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984707115702-26446-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984707115702-26446-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43087
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984707115702-26446-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43087
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:57:33.854045 25026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:57:33.854442 25026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:57:33.854585 25026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:57:33.862316 25026 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:57:33.862447 25026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:57:33.862495 25026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:57:33.862620 25026 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:57:33.862676 25026 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:57:33.873220 25026 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984707115702-26446-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984707115702-26446-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43087
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984707115702-26446-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984707115702-26446-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:57:33.874708 25026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:57:33.876099 25026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:57:33.888098 25033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:33.888053 25038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:33.888015 25036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:57:33.888717 25026 server_base.cc:1034] running on GCE node
I20260806 02:57:33.889583 25026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:57:33.891139 25026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:57:33.892354 25026 hybrid_clock.cc:648] HybridClock initialized: now 1785985053892295 us; error 49 us; skew 500 ppm
I20260806 02:57:33.892689 25026 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:57:33 dist-test-slave-zj8q krb5kdc[25008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985053, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:57:33.896328 25026 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:57:33.898066 25026 webserver.cc:460] Webserver started at http://127.0.0.1:35891/ using document root <none> and password file <none>
I20260806 02:57:33.898833 25026 fs_manager.cc:359] Metadata directory not provided
I20260806 02:57:33.898955 25026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:57:33.899282 25026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:57:33.902544 25026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984707115702-26446-0/minicluster-data/master-0/data/instance:
uuid: "3214ea5db2b14877ab64c4a7a557ca8a"
format_stamp: "Formatted at 2026-08-06 02:57:33 on dist-test-slave-zj8q"
server_key: "46f074524c64fa443bc852358670c3d4"
server_key_iv: "a5dd70b171fa4180c81a68b5d70a7ca2"
server_key_version: "encryptionkey@0"
I20260806 02:57:33.903390 25026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984707115702-26446-0/minicluster-data/master-0/wal/instance:
uuid: "3214ea5db2b14877ab64c4a7a557ca8a"
format_stamp: "Formatted at 2026-08-06 02:57:33 on dist-test-slave-zj8q"
server_key: "46f074524c64fa443bc852358670c3d4"
server_key_iv: "a5dd70b171fa4180c81a68b5d70a7ca2"
server_key_version: "encryptionkey@0"
I20260806 02:57:33.909057 25026 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 02:57:33.912933 25048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:33.914484 25026 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 02:57:33.914683 25026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984707115702-26446-0/minicluster-data/master-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984707115702-26446-0/minicluster-data/master-0/wal
uuid: "3214ea5db2b14877ab64c4a7a557ca8a"
format_stamp: "Formatted at 2026-08-06 02:57:33 on dist-test-slave-zj8q"
server_key: "46f074524c64fa443bc852358670c3d4"
server_key_iv: "a5dd70b171fa4180c81a68b5d70a7ca2"
server_key_version: "encryptionkey@0"
I20260806 02:57:33.914904 25026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984707115702-26446-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984707115702-26446-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984707115702-26446-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:57:33.929294 25026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:57:33.933750 25026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:57:33.934031 25026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:57:33.952100 25026 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43087
I20260806 02:57:33.952119 25127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43087 every 8 connection(s)
I20260806 02:57:33.953528 25026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
I20260806 02:57:33.957451 25128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:57:33.960830 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 25026
I20260806 02:57:33.960973 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1785984707115702-26446-0/minicluster-data/master-0/wal/instance
I20260806 02:57:33.961390 26446 external_mini_cluster.cc:1428] Setting key 6cda5e78664ed06e11e2781fac5ae9fe
I20260806 02:57:33.970736 25128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3214ea5db2b14877ab64c4a7a557ca8a: Bootstrap starting.
I20260806 02:57:33.974835 25128 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3214ea5db2b14877ab64c4a7a557ca8a: Neither blocks nor log segments found. Creating new log.
I20260806 02:57:33.976253 25128 log.cc:824] T 00000000000000000000000000000000 P 3214ea5db2b14877ab64c4a7a557ca8a: Log is configured to *not* fsync() on all Append() calls
I20260806 02:57:33.976426 25132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:33.963246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44023 (local address 127.0.0.1:43087)
0806 02:57:33.964044 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:33.964059 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:33.964112 (+ 53us) server_negotiation.cc:407] Connection header received
0806 02:57:33.966203 (+ 2091us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:33.966233 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:33.966785 (+ 552us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:33.967237 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:33.967752 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:33.969776 (+ 2024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:33.971130 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:33.972152 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:33.972343 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:33.973436 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:33.973447 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:33.973464 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:57:33.974294 (+ 830us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:33.974407 (+ 113us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:57:33.974449 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:33.974577 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:33.974824 (+ 247us) server_negotiation.cc:299] Negotiation successful
0806 02:57:33.975115 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":304,"threads_started":1}
I20260806 02:57:33.979867 25128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3214ea5db2b14877ab64c4a7a557ca8a: No bootstrap required, opened a new log
I20260806 02:57:33.980882 26446 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:57:33.988571 25128 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3214ea5db2b14877ab64c4a7a557ca8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3214ea5db2b14877ab64c4a7a557ca8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43087 } }
I20260806 02:57:33.988963 25128 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3214ea5db2b14877ab64c4a7a557ca8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:57:33.989064 25128 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3214ea5db2b14877ab64c4a7a557ca8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3214ea5db2b14877ab64c4a7a557ca8a, State: Initialized, Role: FOLLOWER
I20260806 02:57:33.989698 25128 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3214ea5db2b14877ab64c4a7a557ca8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3214ea5db2b14877ab64c4a7a557ca8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43087 } }
I20260806 02:57:33.989951 25128 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3214ea5db2b14877ab64c4a7a557ca8a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:57:33.990085 25128 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3214ea5db2b14877ab64c4a7a557ca8a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:57:33.990327 25128 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3214ea5db2b14877ab64c4a7a557ca8a [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:57:33.992089 25132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:33.985452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38651 (local address 127.0.0.1:43087)
0806 02:57:33.985660 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:33.985669 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:33.985955 (+ 286us) server_negotiation.cc:407] Connection header received
0806 02:57:33.986139 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:33.986144 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:33.986245 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:33.986366 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:33.986864 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:33.987959 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:33.989509 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:33.989976 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:33.990112 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:33.991328 (+ 1216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:33.991333 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:33.991336 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:57:33.991355 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:33.991397 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:57:33.991414 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:33.991495 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:33.991682 (+ 187us) server_negotiation.cc:299] Negotiation successful
0806 02:57:33.991792 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260806 02:57:33.993229 25128 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3214ea5db2b14877ab64c4a7a557ca8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3214ea5db2b14877ab64c4a7a557ca8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43087 } }
I20260806 02:57:33.993790 25128 leader_election.cc:302] T 00000000000000000000000000000000 P 3214ea5db2b14877ab64c4a7a557ca8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3214ea5db2b14877ab64c4a7a557ca8a; no voters:
I20260806 02:57:33.994379 25128 leader_election.cc:288] T 00000000000000000000000000000000 P 3214ea5db2b14877ab64c4a7a557ca8a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:57:33.994508 25144 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3214ea5db2b14877ab64c4a7a557ca8a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:57:33.995162 25144 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3214ea5db2b14877ab64c4a7a557ca8a [term 1 LEADER]: Becoming Leader. State: Replica: 3214ea5db2b14877ab64c4a7a557ca8a, State: Running, Role: LEADER
I20260806 02:57:33.995730 25144 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3214ea5db2b14877ab64c4a7a557ca8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3214ea5db2b14877ab64c4a7a557ca8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43087 } }
I20260806 02:57:33.996433 25128 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3214ea5db2b14877ab64c4a7a557ca8a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:57:33.998558 25147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3214ea5db2b14877ab64c4a7a557ca8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3214ea5db2b14877ab64c4a7a557ca8a. Latest consensus state: current_term: 1 leader_uuid: "3214ea5db2b14877ab64c4a7a557ca8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3214ea5db2b14877ab64c4a7a557ca8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43087 } } }
I20260806 02:57:33.999050 25147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3214ea5db2b14877ab64c4a7a557ca8a [sys.catalog]: This master's current role is: LEADER
I20260806 02:57:33.998596 25146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3214ea5db2b14877ab64c4a7a557ca8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3214ea5db2b14877ab64c4a7a557ca8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3214ea5db2b14877ab64c4a7a557ca8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43087 } } }
I20260806 02:57:34.001302 25146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3214ea5db2b14877ab64c4a7a557ca8a [sys.catalog]: This master's current role is: LEADER
I20260806 02:57:34.001421 25154 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:57:34.007019 25154 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:57:34.029377 25154 catalog_manager.cc:1347] Generated new cluster ID: 18dd9b2620194d4fad3671981d13ca79
I20260806 02:57:34.029511 25154 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:57:34.078764 25154 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:57:34.080750 25154 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:57:34.100657 25154 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3214ea5db2b14877ab64c4a7a557ca8a: Generated new TSK 0
I20260806 02:57:34.101797 25154 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:57:34.108587 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 25026
W20260806 02:57:34.109663 25121 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:44023
W20260806 02:57:34.110392 26446 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 25026: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 02:57:34.110543 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 25026
2026-08-06T02:57:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2183 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984707115702-26446-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:57:34 dist-test-slave-zj8q krb5kdc[25186](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:57:34 dist-test-slave-zj8q krb5kdc[25186](info): set up 2 sockets
Aug 06 02:57:34 dist-test-slave-zj8q krb5kdc[25186](info): commencing operation
krb5kdc: starting...
W20260806 02:57:35.329275 26446 mini_kdc.cc:121] Time spent starting KDC: real 1.177s user 0.005s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Aug 06 02:57:35 dist-test-slave-zj8q krb5kdc[25186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:57:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:57:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984707115702-26446-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984707115702-26446-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:57:35.673538 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984707115702-26446-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984707115702-26446-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984707115702-26446-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41219
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984707115702-26446-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41219
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:57:36.042232 25205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:57:36.042583 25205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:57:36.042737 25205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:57:36.050336 25205 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:57:36.050472 25205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:57:36.050547 25205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:57:36.050637 25205 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:57:36.050693 25205 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:57:36.061183 25205 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984707115702-26446-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984707115702-26446-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984707115702-26446-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984707115702-26446-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:57:36.062662 25205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:57:36.064059 25205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:57:36.076417 25213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:36.076351 25216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:36.076334 25214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:57:36.077060 25205 server_base.cc:1034] running on GCE node
I20260806 02:57:36.078028 25205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:57:36.079725 25205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:57:36.080961 25205 hybrid_clock.cc:648] HybridClock initialized: now 1785985056080911 us; error 44 us; skew 500 ppm
I20260806 02:57:36.081354 25205 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:57:36 dist-test-slave-zj8q krb5kdc[25186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985056, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:57:36.084703 25205 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:57:36.086445 25205 webserver.cc:460] Webserver started at http://127.0.0.1:39559/ using document root <none> and password file <none>
I20260806 02:57:36.087173 25205 fs_manager.cc:359] Metadata directory not provided
I20260806 02:57:36.087301 25205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:57:36.087608 25205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:57:36.090874 25205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984707115702-26446-0/minicluster-data/master-0/data/instance:
uuid: "5476ae65893b42e295728ec1162d864a"
format_stamp: "Formatted at 2026-08-06 02:57:36 on dist-test-slave-zj8q"
server_key: "bfa8699e40bc27ed6eaeec7788dc3d6b"
server_key_iv: "2ddf3ee3987e9cdeca11bebba6f2c50d"
server_key_version: "encryptionkey@0"
I20260806 02:57:36.091712 25205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984707115702-26446-0/minicluster-data/master-0/wal/instance:
uuid: "5476ae65893b42e295728ec1162d864a"
format_stamp: "Formatted at 2026-08-06 02:57:36 on dist-test-slave-zj8q"
server_key: "bfa8699e40bc27ed6eaeec7788dc3d6b"
server_key_iv: "2ddf3ee3987e9cdeca11bebba6f2c50d"
server_key_version: "encryptionkey@0"
I20260806 02:57:36.097373 25205 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260806 02:57:36.101276 25229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:36.102881 25205 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:57:36.103089 25205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984707115702-26446-0/minicluster-data/master-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984707115702-26446-0/minicluster-data/master-0/wal
uuid: "5476ae65893b42e295728ec1162d864a"
format_stamp: "Formatted at 2026-08-06 02:57:36 on dist-test-slave-zj8q"
server_key: "bfa8699e40bc27ed6eaeec7788dc3d6b"
server_key_iv: "2ddf3ee3987e9cdeca11bebba6f2c50d"
server_key_version: "encryptionkey@0"
I20260806 02:57:36.103308 25205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984707115702-26446-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984707115702-26446-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984707115702-26446-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:57:36.109740 25205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:57:36.114377 25205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:57:36.114682 25205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:57:36.133910 25205 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41219
I20260806 02:57:36.133952 25306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41219 every 8 connection(s)
I20260806 02:57:36.135401 25205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
I20260806 02:57:36.137633 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 25205
I20260806 02:57:36.137825 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1785984707115702-26446-0/minicluster-data/master-0/wal/instance
I20260806 02:57:36.138330 26446 external_mini_cluster.cc:1428] Setting key 958243b46a960dc74484c65da2f61741
I20260806 02:57:36.139611 25308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:57:36.147464 25311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:36.140323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48985 (local address 127.0.0.1:41219)
0806 02:57:36.141215 (+ 892us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:36.141244 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:36.141301 (+ 57us) server_negotiation.cc:407] Connection header received
0806 02:57:36.143241 (+ 1940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:36.143271 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:36.143788 (+ 517us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:36.144289 (+ 501us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:36.145315 (+ 1026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:36.145323 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:36.145340 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:57:36.145357 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:36.145453 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:57:36.145481 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:36.145737 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:36.145884 (+ 147us) server_negotiation.cc:299] Negotiation successful
0806 02:57:36.146105 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":368,"threads_started":1}
I20260806 02:57:36.151557 26446 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:57:36.154910 25308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5476ae65893b42e295728ec1162d864a: Bootstrap starting.
I20260806 02:57:36.160657 25311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:36.157742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60325 (local address 127.0.0.1:41219)
0806 02:57:36.157936 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:36.157943 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:36.158313 (+ 370us) server_negotiation.cc:407] Connection header received
0806 02:57:36.158522 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:36.158528 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:36.158602 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:36.158708 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:36.159779 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:36.159785 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:36.159790 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:57:36.159811 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:36.159863 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:57:36.159884 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:36.160002 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:36.160115 (+ 113us) server_negotiation.cc:299] Negotiation successful
0806 02:57:36.160193 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 02:57:36.161365 25308 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5476ae65893b42e295728ec1162d864a: Neither blocks nor log segments found. Creating new log.
I20260806 02:57:36.163111 25308 log.cc:824] T 00000000000000000000000000000000 P 5476ae65893b42e295728ec1162d864a: Log is configured to *not* fsync() on all Append() calls
I20260806 02:57:36.168442 25308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5476ae65893b42e295728ec1162d864a: No bootstrap required, opened a new log
I20260806 02:57:36.176185 25308 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5476ae65893b42e295728ec1162d864a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5476ae65893b42e295728ec1162d864a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41219 } }
I20260806 02:57:36.176707 25308 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5476ae65893b42e295728ec1162d864a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:57:36.176832 25308 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5476ae65893b42e295728ec1162d864a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5476ae65893b42e295728ec1162d864a, State: Initialized, Role: FOLLOWER
I20260806 02:57:36.177452 25308 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5476ae65893b42e295728ec1162d864a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5476ae65893b42e295728ec1162d864a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41219 } }
I20260806 02:57:36.177670 25308 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5476ae65893b42e295728ec1162d864a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:57:36.177804 25308 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5476ae65893b42e295728ec1162d864a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:57:36.177975 25308 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5476ae65893b42e295728ec1162d864a [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:57:36.180616 25308 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5476ae65893b42e295728ec1162d864a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5476ae65893b42e295728ec1162d864a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41219 } }
I20260806 02:57:36.181154 25308 leader_election.cc:302] T 00000000000000000000000000000000 P 5476ae65893b42e295728ec1162d864a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5476ae65893b42e295728ec1162d864a; no voters:
I20260806 02:57:36.181839 25308 leader_election.cc:288] T 00000000000000000000000000000000 P 5476ae65893b42e295728ec1162d864a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:57:36.181990 25320 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5476ae65893b42e295728ec1162d864a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:57:36.182719 25320 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5476ae65893b42e295728ec1162d864a [term 1 LEADER]: Becoming Leader. State: Replica: 5476ae65893b42e295728ec1162d864a, State: Running, Role: LEADER
I20260806 02:57:36.183318 25320 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5476ae65893b42e295728ec1162d864a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5476ae65893b42e295728ec1162d864a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41219 } }
I20260806 02:57:36.183996 25308 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5476ae65893b42e295728ec1162d864a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:57:36.185913 25322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5476ae65893b42e295728ec1162d864a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5476ae65893b42e295728ec1162d864a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5476ae65893b42e295728ec1162d864a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41219 } } }
I20260806 02:57:36.186304 25322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5476ae65893b42e295728ec1162d864a [sys.catalog]: This master's current role is: LEADER
I20260806 02:57:36.186367 25324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5476ae65893b42e295728ec1162d864a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5476ae65893b42e295728ec1162d864a. Latest consensus state: current_term: 1 leader_uuid: "5476ae65893b42e295728ec1162d864a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5476ae65893b42e295728ec1162d864a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41219 } } }
I20260806 02:57:36.186669 25324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5476ae65893b42e295728ec1162d864a [sys.catalog]: This master's current role is: LEADER
I20260806 02:57:36.190011 25334 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:57:36.195366 25334 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:57:36.206209 25334 catalog_manager.cc:1347] Generated new cluster ID: 39d9cf3eef3445a09df6bd3b99882bd0
I20260806 02:57:36.206347 25334 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:57:36.229399 25334 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:57:36.229846 25334 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:57:36.238597 25334 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5476ae65893b42e295728ec1162d864a: Generated new TSK 0
I20260806 02:57:36.239634 25334 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:57:36.259584 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 25205
W20260806 02:57:36.261049 25300 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48985
W20260806 02:57:36.261924 26446 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 25205: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 02:57:36.262072 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 25205
2026-08-06T02:57:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2157 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984707115702-26446-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:57:36 dist-test-slave-zj8q krb5kdc[25353](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:57:36 dist-test-slave-zj8q krb5kdc[25353](info): set up 2 sockets
Aug 06 02:57:36 dist-test-slave-zj8q krb5kdc[25353](info): commencing operation
krb5kdc: starting...
W20260806 02:57:37.499776 26446 mini_kdc.cc:121] Time spent starting KDC: real 1.189s user 0.006s sys 0.046s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Aug 06 02:57:37 dist-test-slave-zj8q krb5kdc[25353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:57:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:57:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984707115702-26446-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984707115702-26446-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:57:37.845127 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984707115702-26446-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984707115702-26446-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984707115702-26446-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32857
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984707115702-26446-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32857
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:57:38.201403 25379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:57:38.201730 25379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:57:38.201872 25379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:57:38.209213 25379 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:57:38.209353 25379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:57:38.209439 25379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:57:38.209524 25379 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:57:38.209586 25379 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:57:38.219775 25379 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984707115702-26446-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984707115702-26446-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32857
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984707115702-26446-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984707115702-26446-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:57:38.221168 25379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:57:38.222483 25379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:57:38.234020 25387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:38.234036 25388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:38.234174 25391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:57:38.234906 25379 server_base.cc:1034] running on GCE node
I20260806 02:57:38.235729 25379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:57:38.237201 25379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:57:38.238415 25379 hybrid_clock.cc:648] HybridClock initialized: now 1785985058238362 us; error 50 us; skew 500 ppm
I20260806 02:57:38.238801 25379 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:57:38 dist-test-slave-zj8q krb5kdc[25353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985058, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:57:38.241938 25379 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:57:38.243561 25379 webserver.cc:460] Webserver started at http://127.0.0.1:37489/ using document root <none> and password file <none>
I20260806 02:57:38.244380 25379 fs_manager.cc:359] Metadata directory not provided
I20260806 02:57:38.244508 25379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:57:38.244815 25379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:57:38.248148 25379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984707115702-26446-0/minicluster-data/master-0/data/instance:
uuid: "4ee509b3391740e9b48f33299c75d509"
format_stamp: "Formatted at 2026-08-06 02:57:38 on dist-test-slave-zj8q"
server_key: "cff0ab61c6d27f60ee46f4bcc33cf47d"
server_key_iv: "6828e7d8f312661c8c4b99c9f9f70e2d"
server_key_version: "encryptionkey@0"
I20260806 02:57:38.248991 25379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984707115702-26446-0/minicluster-data/master-0/wal/instance:
uuid: "4ee509b3391740e9b48f33299c75d509"
format_stamp: "Formatted at 2026-08-06 02:57:38 on dist-test-slave-zj8q"
server_key: "cff0ab61c6d27f60ee46f4bcc33cf47d"
server_key_iv: "6828e7d8f312661c8c4b99c9f9f70e2d"
server_key_version: "encryptionkey@0"
I20260806 02:57:38.254868 25379 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260806 02:57:38.258636 25402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:38.260100 25379 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 02:57:38.260303 25379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984707115702-26446-0/minicluster-data/master-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984707115702-26446-0/minicluster-data/master-0/wal
uuid: "4ee509b3391740e9b48f33299c75d509"
format_stamp: "Formatted at 2026-08-06 02:57:38 on dist-test-slave-zj8q"
server_key: "cff0ab61c6d27f60ee46f4bcc33cf47d"
server_key_iv: "6828e7d8f312661c8c4b99c9f9f70e2d"
server_key_version: "encryptionkey@0"
I20260806 02:57:38.260506 25379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984707115702-26446-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984707115702-26446-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984707115702-26446-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:57:38.290890 25379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:57:38.295614 25379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:57:38.295933 25379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:57:38.314656 25379 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32857
I20260806 02:57:38.314697 25481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32857 every 8 connection(s)
I20260806 02:57:38.316080 25379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
I20260806 02:57:38.319770 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 25379
I20260806 02:57:38.319968 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1785984707115702-26446-0/minicluster-data/master-0/wal/instance
I20260806 02:57:38.320142 25483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:57:38.320549 26446 external_mini_cluster.cc:1428] Setting key e5da814becf8554ac46cde96e916de57
I20260806 02:57:38.335429 25483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ee509b3391740e9b48f33299c75d509: Bootstrap starting.
Aug 06 02:57:38 dist-test-slave-zj8q krb5kdc[25353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 02:57:38.339187 25483 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ee509b3391740e9b48f33299c75d509: Neither blocks nor log segments found. Creating new log.
I20260806 02:57:38.340323 25483 log.cc:824] T 00000000000000000000000000000000 P 4ee509b3391740e9b48f33299c75d509: Log is configured to *not* fsync() on all Append() calls
I20260806 02:57:38.343469 25483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ee509b3391740e9b48f33299c75d509: No bootstrap required, opened a new log
I20260806 02:57:38.344591 25487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:38.322574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.236:34861 (local address 127.0.0.1:32857)
0806 02:57:38.323425 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:38.323436 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:38.323479 (+ 43us) server_negotiation.cc:407] Connection header received
0806 02:57:38.325129 (+ 1650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:38.325153 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:38.325609 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:38.325952 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:38.327861 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:38.329206 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:38.331044 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:38.331810 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:38.331955 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:38.336851 (+ 4896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:38.336859 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:38.336874 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:57:38.336927 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:38.340127 (+ 3200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:38.341055 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:38.341066 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:38.341074 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:38.341277 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:38.341756 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:38.341760 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:38.341762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:38.342379 (+ 617us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:38.342683 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:38.343217 (+ 534us) server_negotiation.cc:299] Negotiation successful
0806 02:57:38.343537 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":337,"threads_started":1}
I20260806 02:57:38.348665 26446 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:57:38.351874 25483 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ee509b3391740e9b48f33299c75d509 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ee509b3391740e9b48f33299c75d509" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32857 } }
I20260806 02:57:38.352324 25483 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ee509b3391740e9b48f33299c75d509 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:57:38.352424 25483 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ee509b3391740e9b48f33299c75d509 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ee509b3391740e9b48f33299c75d509, State: Initialized, Role: FOLLOWER
I20260806 02:57:38.353181 25483 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ee509b3391740e9b48f33299c75d509 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ee509b3391740e9b48f33299c75d509" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32857 } }
I20260806 02:57:38.353500 25483 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ee509b3391740e9b48f33299c75d509 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:57:38.353664 25483 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ee509b3391740e9b48f33299c75d509 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:57:38.353843 25483 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ee509b3391740e9b48f33299c75d509 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:57:38.356200 25483 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ee509b3391740e9b48f33299c75d509 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ee509b3391740e9b48f33299c75d509" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32857 } }
I20260806 02:57:38.356738 25483 leader_election.cc:302] T 00000000000000000000000000000000 P 4ee509b3391740e9b48f33299c75d509 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ee509b3391740e9b48f33299c75d509; no voters:
I20260806 02:57:38.357366 25483 leader_election.cc:288] T 00000000000000000000000000000000 P 4ee509b3391740e9b48f33299c75d509 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:57:38.357473 25498 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ee509b3391740e9b48f33299c75d509 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:57:38.359896 25483 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ee509b3391740e9b48f33299c75d509 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:57:38.364410 25498 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ee509b3391740e9b48f33299c75d509 [term 1 LEADER]: Becoming Leader. State: Replica: 4ee509b3391740e9b48f33299c75d509, State: Running, Role: LEADER
I20260806 02:57:38.365213 25498 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ee509b3391740e9b48f33299c75d509 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ee509b3391740e9b48f33299c75d509" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32857 } }
I20260806 02:57:38.368311 25487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:38.353088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.236:59071 (local address 127.0.0.1:32857)
0806 02:57:38.353255 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:38.353261 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:38.353743 (+ 482us) server_negotiation.cc:407] Connection header received
0806 02:57:38.353953 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:38.353957 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:38.354038 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:38.354138 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:38.355630 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:38.356407 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:38.357727 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:38.358464 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:38.358596 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:38.361083 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:38.361088 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:38.361091 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:57:38.361124 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:38.364284 (+ 3160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:38.365078 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:38.365083 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:38.365086 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:38.365245 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:38.365879 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:38.365885 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:38.365889 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:38.366364 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:38.367648 (+ 1284us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:38.367812 (+ 164us) server_negotiation.cc:299] Negotiation successful
0806 02:57:38.367948 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 02:57:38.374430 25500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ee509b3391740e9b48f33299c75d509 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ee509b3391740e9b48f33299c75d509. Latest consensus state: current_term: 1 leader_uuid: "4ee509b3391740e9b48f33299c75d509" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ee509b3391740e9b48f33299c75d509" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32857 } } }
I20260806 02:57:38.374854 25500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ee509b3391740e9b48f33299c75d509 [sys.catalog]: This master's current role is: LEADER
I20260806 02:57:38.375362 25499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ee509b3391740e9b48f33299c75d509 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ee509b3391740e9b48f33299c75d509" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ee509b3391740e9b48f33299c75d509" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32857 } } }
I20260806 02:57:38.375655 25499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ee509b3391740e9b48f33299c75d509 [sys.catalog]: This master's current role is: LEADER
I20260806 02:57:38.376543 25512 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:57:38.381870 25512 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:57:38.391625 25512 catalog_manager.cc:1347] Generated new cluster ID: 04a56e2808f74980b30ca93b05a7d5a1
I20260806 02:57:38.391776 25512 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:57:38.400646 25512 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:57:38.402619 25512 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:57:38.410554 25512 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ee509b3391740e9b48f33299c75d509: Generated new TSK 0
I20260806 02:57:38.411490 25512 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:57:38.426774 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 25379
I20260806 02:57:38.428292 25476 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:57:38.519779 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 25379
W20260806 02:57:38.543457 25350 connection.cc:535] client connection to 127.0.0.1:32857 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32857): Connection reset by peer (error 104)
2026-08-06T02:57:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2252 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984707115702-26446-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:57:38 dist-test-slave-zj8q krb5kdc[25523](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:57:38 dist-test-slave-zj8q krb5kdc[25523](info): set up 2 sockets
Aug 06 02:57:38 dist-test-slave-zj8q krb5kdc[25523](info): commencing operation
krb5kdc: starting...
W20260806 02:57:39.749423 26446 mini_kdc.cc:121] Time spent starting KDC: real 1.187s user 0.000s sys 0.050s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Aug 06 02:57:39 dist-test-slave-zj8q krb5kdc[25523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:57:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:57:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984707115702-26446-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984707115702-26446-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:57:40.089586 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984707115702-26446-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984707115702-26446-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984707115702-26446-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42313
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984707115702-26446-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42313
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:57:40.440297 25546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:57:40.440649 25546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:57:40.440796 25546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:57:40.448313 25546 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:57:40.448482 25546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:57:40.448577 25546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:57:40.448660 25546 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:57:40.448721 25546 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:57:40.459187 25546 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984707115702-26446-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984707115702-26446-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984707115702-26446-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984707115702-26446-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:57:40.460558 25546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:57:40.462009 25546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:57:40.474100 25554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:40.474138 25555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:40.474059 25558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:57:40.475214 25546 server_base.cc:1034] running on GCE node
I20260806 02:57:40.476032 25546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:57:40.477491 25546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:57:40.478688 25546 hybrid_clock.cc:648] HybridClock initialized: now 1785985060478639 us; error 42 us; skew 500 ppm
I20260806 02:57:40.479040 25546 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:57:40 dist-test-slave-zj8q krb5kdc[25523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985060, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:57:40.482069 25546 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:57:40.483525 25546 webserver.cc:460] Webserver started at http://127.0.0.1:36553/ using document root <none> and password file <none>
I20260806 02:57:40.484211 25546 fs_manager.cc:359] Metadata directory not provided
I20260806 02:57:40.484333 25546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:57:40.484624 25546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:57:40.487895 25546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984707115702-26446-0/minicluster-data/master-0/data/instance:
uuid: "6c152ff913534df5aaebcb0ca1f46625"
format_stamp: "Formatted at 2026-08-06 02:57:40 on dist-test-slave-zj8q"
server_key: "9d4585044325c6e5bdbac5b8e4dc534d"
server_key_iv: "922301a9ae112dd5d5ca5f34ae2c0f91"
server_key_version: "encryptionkey@0"
I20260806 02:57:40.488689 25546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984707115702-26446-0/minicluster-data/master-0/wal/instance:
uuid: "6c152ff913534df5aaebcb0ca1f46625"
format_stamp: "Formatted at 2026-08-06 02:57:40 on dist-test-slave-zj8q"
server_key: "9d4585044325c6e5bdbac5b8e4dc534d"
server_key_iv: "922301a9ae112dd5d5ca5f34ae2c0f91"
server_key_version: "encryptionkey@0"
I20260806 02:57:40.494004 25546 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260806 02:57:40.497520 25571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:40.498867 25546 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:57:40.499040 25546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984707115702-26446-0/minicluster-data/master-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984707115702-26446-0/minicluster-data/master-0/wal
uuid: "6c152ff913534df5aaebcb0ca1f46625"
format_stamp: "Formatted at 2026-08-06 02:57:40 on dist-test-slave-zj8q"
server_key: "9d4585044325c6e5bdbac5b8e4dc534d"
server_key_iv: "922301a9ae112dd5d5ca5f34ae2c0f91"
server_key_version: "encryptionkey@0"
I20260806 02:57:40.499228 25546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984707115702-26446-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984707115702-26446-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984707115702-26446-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:57:40.528231 25546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:57:40.532507 25546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:57:40.532804 25546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:57:40.551436 25546 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42313
I20260806 02:57:40.551469 25640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42313 every 8 connection(s)
I20260806 02:57:40.552814 25546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
I20260806 02:57:40.553736 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 25546
I20260806 02:57:40.553929 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1785984707115702-26446-0/minicluster-data/master-0/wal/instance
I20260806 02:57:40.554502 26446 external_mini_cluster.cc:1428] Setting key b76faf2e690feccf9790ef92cef67967
I20260806 02:57:40.557374 25642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:57:40.568255 25646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:40.556390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.236:56703 (local address 127.0.0.1:42313)
0806 02:57:40.557191 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:40.557203 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:40.557244 (+ 41us) server_negotiation.cc:407] Connection header received
0806 02:57:40.558953 (+ 1709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:40.558977 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:40.559348 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:40.559666 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:40.560276 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:40.561702 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:40.563579 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:40.564267 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:40.564402 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:40.565755 (+ 1353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:40.565771 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:40.565786 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:57:40.566423 (+ 637us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:40.566514 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:57:40.566544 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:40.566680 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:40.566970 (+ 290us) server_negotiation.cc:299] Negotiation successful
0806 02:57:40.567183 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":367,"threads_started":1}
I20260806 02:57:40.569758 25642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c152ff913534df5aaebcb0ca1f46625: Bootstrap starting.
I20260806 02:57:40.572180 26446 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:57:40.573261 25642 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c152ff913534df5aaebcb0ca1f46625: Neither blocks nor log segments found. Creating new log.
I20260806 02:57:40.574554 25642 log.cc:824] T 00000000000000000000000000000000 P 6c152ff913534df5aaebcb0ca1f46625: Log is configured to *not* fsync() on all Append() calls
I20260806 02:57:40.578078 25642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c152ff913534df5aaebcb0ca1f46625: No bootstrap required, opened a new log
I20260806 02:57:40.583900 25646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:40.577560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.236:57913 (local address 127.0.0.1:42313)
0806 02:57:40.577747 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:40.577753 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:40.578239 (+ 486us) server_negotiation.cc:407] Connection header received
0806 02:57:40.578506 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:40.578510 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:40.578579 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:40.578663 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:40.579188 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:40.579998 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:40.581407 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:40.581871 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:40.582011 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:40.583141 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:40.583146 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:40.583149 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:57:40.583169 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:40.583229 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:57:40.583245 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:40.583350 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:40.583559 (+ 209us) server_negotiation.cc:299] Negotiation successful
0806 02:57:40.583616 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 02:57:40.586093 25642 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c152ff913534df5aaebcb0ca1f46625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c152ff913534df5aaebcb0ca1f46625" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42313 } }
I20260806 02:57:40.586555 25642 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c152ff913534df5aaebcb0ca1f46625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:57:40.586651 25642 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c152ff913534df5aaebcb0ca1f46625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c152ff913534df5aaebcb0ca1f46625, State: Initialized, Role: FOLLOWER
I20260806 02:57:40.587198 25642 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c152ff913534df5aaebcb0ca1f46625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c152ff913534df5aaebcb0ca1f46625" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42313 } }
I20260806 02:57:40.587500 25642 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c152ff913534df5aaebcb0ca1f46625 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:57:40.587644 25642 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c152ff913534df5aaebcb0ca1f46625 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:57:40.587787 25642 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c152ff913534df5aaebcb0ca1f46625 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:57:40.590313 25642 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c152ff913534df5aaebcb0ca1f46625 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c152ff913534df5aaebcb0ca1f46625" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42313 } }
I20260806 02:57:40.590795 25642 leader_election.cc:302] T 00000000000000000000000000000000 P 6c152ff913534df5aaebcb0ca1f46625 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c152ff913534df5aaebcb0ca1f46625; no voters:
I20260806 02:57:40.591356 25642 leader_election.cc:288] T 00000000000000000000000000000000 P 6c152ff913534df5aaebcb0ca1f46625 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:57:40.591487 25656 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c152ff913534df5aaebcb0ca1f46625 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:57:40.592804 25656 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c152ff913534df5aaebcb0ca1f46625 [term 1 LEADER]: Becoming Leader. State: Replica: 6c152ff913534df5aaebcb0ca1f46625, State: Running, Role: LEADER
I20260806 02:57:40.593559 25642 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c152ff913534df5aaebcb0ca1f46625 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:57:40.593475 25656 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c152ff913534df5aaebcb0ca1f46625 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c152ff913534df5aaebcb0ca1f46625" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42313 } }
I20260806 02:57:40.596410 25660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c152ff913534df5aaebcb0ca1f46625 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c152ff913534df5aaebcb0ca1f46625" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c152ff913534df5aaebcb0ca1f46625" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42313 } } }
I20260806 02:57:40.596436 25661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c152ff913534df5aaebcb0ca1f46625 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c152ff913534df5aaebcb0ca1f46625. Latest consensus state: current_term: 1 leader_uuid: "6c152ff913534df5aaebcb0ca1f46625" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c152ff913534df5aaebcb0ca1f46625" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42313 } } }
I20260806 02:57:40.597121 25661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c152ff913534df5aaebcb0ca1f46625 [sys.catalog]: This master's current role is: LEADER
I20260806 02:57:40.597121 25660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c152ff913534df5aaebcb0ca1f46625 [sys.catalog]: This master's current role is: LEADER
I20260806 02:57:40.598771 25671 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:57:40.604569 25671 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:57:40.615586 25671 catalog_manager.cc:1347] Generated new cluster ID: ecf49aea34cf41f9b2ddbfcff8f721a8
I20260806 02:57:40.615746 25671 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:57:40.651165 25671 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:57:40.653142 25671 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:57:40.667209 25671 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c152ff913534df5aaebcb0ca1f46625: Generated new TSK 0
I20260806 02:57:40.668396 25671 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:57:40.686496 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 25546
W20260806 02:57:40.687724 25635 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.236:56703
W20260806 02:57:40.688617 26446 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 25546: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 02:57:40.688771 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 25546
W20260806 02:57:40.711369 25520 connection.cc:535] client connection to 127.0.0.1:42313 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42313): Connection reset by peer (error 104)
2026-08-06T02:57:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2167 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984707115702-26446-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:57:40 dist-test-slave-zj8q krb5kdc[25689](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:57:40 dist-test-slave-zj8q krb5kdc[25689](info): set up 2 sockets
Aug 06 02:57:40 dist-test-slave-zj8q krb5kdc[25689](info): commencing operation
krb5kdc: starting...
W20260806 02:57:41.931877 26446 mini_kdc.cc:121] Time spent starting KDC: real 1.201s user 0.000s sys 0.052s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Aug 06 02:57:42 dist-test-slave-zj8q krb5kdc[25689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985062, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:57:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:57:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984707115702-26446-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984707115702-26446-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:57:42.277740 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984707115702-26446-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984707115702-26446-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984707115702-26446-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46333
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984707115702-26446-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46333
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:57:42.633879 25712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:57:42.634258 25712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:57:42.634411 25712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:57:42.642239 25712 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:57:42.642367 25712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:57:42.642423 25712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:57:42.642508 25712 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:57:42.642570 25712 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:57:42.652774 25712 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984707115702-26446-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984707115702-26446-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46333
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984707115702-26446-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46333
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984707115702-26446-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:57:42.654117 25712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:57:42.655522 25712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:57:42.667366 25727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:42.667454 25725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:42.667488 25724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:57:42.668292 25712 server_base.cc:1034] running on GCE node
I20260806 02:57:42.669111 25712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:57:42.670632 25712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:57:42.671859 25712 hybrid_clock.cc:648] HybridClock initialized: now 1785985062671808 us; error 45 us; skew 500 ppm
I20260806 02:57:42.672209 25712 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:57:42 dist-test-slave-zj8q krb5kdc[25689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985062, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:57:42.679999 25712 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:57:42.681684 25712 webserver.cc:460] Webserver started at http://127.0.0.1:42941/ using document root <none> and password file <none>
I20260806 02:57:42.682476 25712 fs_manager.cc:359] Metadata directory not provided
I20260806 02:57:42.682610 25712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:57:42.682919 25712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:57:42.686272 25712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984707115702-26446-0/minicluster-data/master-0/data/instance:
uuid: "226d379961e94e1a9332f7e1805c20f9"
format_stamp: "Formatted at 2026-08-06 02:57:42 on dist-test-slave-zj8q"
server_key: "46f9a6487db6a95db03bdfe2a37e6a5b"
server_key_iv: "050b4a1cc719ef56877b6c47d994aeab"
server_key_version: "encryptionkey@0"
I20260806 02:57:42.687134 25712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984707115702-26446-0/minicluster-data/master-0/wal/instance:
uuid: "226d379961e94e1a9332f7e1805c20f9"
format_stamp: "Formatted at 2026-08-06 02:57:42 on dist-test-slave-zj8q"
server_key: "46f9a6487db6a95db03bdfe2a37e6a5b"
server_key_iv: "050b4a1cc719ef56877b6c47d994aeab"
server_key_version: "encryptionkey@0"
I20260806 02:57:42.692787 25712 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260806 02:57:42.696568 25735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:42.698024 25712 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 02:57:42.698251 25712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984707115702-26446-0/minicluster-data/master-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984707115702-26446-0/minicluster-data/master-0/wal
uuid: "226d379961e94e1a9332f7e1805c20f9"
format_stamp: "Formatted at 2026-08-06 02:57:42 on dist-test-slave-zj8q"
server_key: "46f9a6487db6a95db03bdfe2a37e6a5b"
server_key_iv: "050b4a1cc719ef56877b6c47d994aeab"
server_key_version: "encryptionkey@0"
I20260806 02:57:42.698455 25712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984707115702-26446-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984707115702-26446-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984707115702-26446-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:57:42.704545 25712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:57:42.708716 25712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:57:42.709012 25712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:57:42.727160 25712 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46333
I20260806 02:57:42.727214 25811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46333 every 8 connection(s)
I20260806 02:57:42.728561 25712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
I20260806 02:57:42.732021 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 25712
I20260806 02:57:42.732182 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1785984707115702-26446-0/minicluster-data/master-0/wal/instance
I20260806 02:57:42.732527 25815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:57:42.732642 26446 external_mini_cluster.cc:1428] Setting key 6cd38c62579c83779a11f5c889544071
I20260806 02:57:42.740705 25822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:42.734470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.236:34521 (local address 127.0.0.1:46333)
0806 02:57:42.735266 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:42.735278 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:42.735324 (+ 46us) server_negotiation.cc:407] Connection header received
0806 02:57:42.737238 (+ 1914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:42.737262 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:42.737615 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:42.737943 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:42.739123 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:42.739130 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:42.739143 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:57:42.739157 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:42.739227 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:57:42.739248 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:42.739421 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:42.739531 (+ 110us) server_negotiation.cc:299] Negotiation successful
0806 02:57:42.739720 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":338,"threads_started":1}
I20260806 02:57:42.744812 26446 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:57:42.746673 25815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 226d379961e94e1a9332f7e1805c20f9: Bootstrap starting.
I20260806 02:57:42.750475 25815 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 226d379961e94e1a9332f7e1805c20f9: Neither blocks nor log segments found. Creating new log.
I20260806 02:57:42.751596 25815 log.cc:824] T 00000000000000000000000000000000 P 226d379961e94e1a9332f7e1805c20f9: Log is configured to *not* fsync() on all Append() calls
I20260806 02:57:42.752056 25822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:42.749540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.236:44423 (local address 127.0.0.1:46333)
0806 02:57:42.749726 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:42.749733 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:42.750136 (+ 403us) server_negotiation.cc:407] Connection header received
0806 02:57:42.750351 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:42.750355 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:42.750414 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:42.750507 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:42.751413 (+ 906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:42.751419 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:42.751422 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:57:42.751442 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:42.751483 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:57:42.751499 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:42.751604 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:42.751718 (+ 114us) server_negotiation.cc:299] Negotiation successful
0806 02:57:42.751783 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 02:57:42.754889 25815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 226d379961e94e1a9332f7e1805c20f9: No bootstrap required, opened a new log
I20260806 02:57:42.762502 25815 raft_consensus.cc:348] T 00000000000000000000000000000000 P 226d379961e94e1a9332f7e1805c20f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "226d379961e94e1a9332f7e1805c20f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46333 } }
I20260806 02:57:42.762907 25815 raft_consensus.cc:374] T 00000000000000000000000000000000 P 226d379961e94e1a9332f7e1805c20f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:57:42.763012 25815 raft_consensus.cc:729] T 00000000000000000000000000000000 P 226d379961e94e1a9332f7e1805c20f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 226d379961e94e1a9332f7e1805c20f9, State: Initialized, Role: FOLLOWER
I20260806 02:57:42.763653 25815 consensus_queue.cc:260] T 00000000000000000000000000000000 P 226d379961e94e1a9332f7e1805c20f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "226d379961e94e1a9332f7e1805c20f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46333 } }
I20260806 02:57:42.763927 25815 raft_consensus.cc:388] T 00000000000000000000000000000000 P 226d379961e94e1a9332f7e1805c20f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:57:42.764083 25815 raft_consensus.cc:482] T 00000000000000000000000000000000 P 226d379961e94e1a9332f7e1805c20f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:57:42.764219 25815 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 226d379961e94e1a9332f7e1805c20f9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:57:42.766407 25815 raft_consensus.cc:504] T 00000000000000000000000000000000 P 226d379961e94e1a9332f7e1805c20f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "226d379961e94e1a9332f7e1805c20f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46333 } }
I20260806 02:57:42.766870 25815 leader_election.cc:302] T 00000000000000000000000000000000 P 226d379961e94e1a9332f7e1805c20f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 226d379961e94e1a9332f7e1805c20f9; no voters:
I20260806 02:57:42.767416 25815 leader_election.cc:288] T 00000000000000000000000000000000 P 226d379961e94e1a9332f7e1805c20f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:57:42.767544 25832 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 226d379961e94e1a9332f7e1805c20f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:57:42.768168 25832 raft_consensus.cc:686] T 00000000000000000000000000000000 P 226d379961e94e1a9332f7e1805c20f9 [term 1 LEADER]: Becoming Leader. State: Replica: 226d379961e94e1a9332f7e1805c20f9, State: Running, Role: LEADER
I20260806 02:57:42.768700 25832 consensus_queue.cc:237] T 00000000000000000000000000000000 P 226d379961e94e1a9332f7e1805c20f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "226d379961e94e1a9332f7e1805c20f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46333 } }
I20260806 02:57:42.769438 25815 sys_catalog.cc:564] T 00000000000000000000000000000000 P 226d379961e94e1a9332f7e1805c20f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:57:42.771188 25834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 226d379961e94e1a9332f7e1805c20f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "226d379961e94e1a9332f7e1805c20f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "226d379961e94e1a9332f7e1805c20f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46333 } } }
I20260806 02:57:42.771617 25834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 226d379961e94e1a9332f7e1805c20f9 [sys.catalog]: This master's current role is: LEADER
I20260806 02:57:42.771863 25837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 226d379961e94e1a9332f7e1805c20f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 226d379961e94e1a9332f7e1805c20f9. Latest consensus state: current_term: 1 leader_uuid: "226d379961e94e1a9332f7e1805c20f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "226d379961e94e1a9332f7e1805c20f9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46333 } } }
I20260806 02:57:42.773062 25837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 226d379961e94e1a9332f7e1805c20f9 [sys.catalog]: This master's current role is: LEADER
I20260806 02:57:42.773808 25841 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:57:42.779523 25841 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:57:42.788586 25841 catalog_manager.cc:1347] Generated new cluster ID: 74137df0ecce48ecba6923f5bd0a2d02
I20260806 02:57:42.788719 25841 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:57:42.801206 25841 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:57:42.801676 25841 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:57:42.814743 25841 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 226d379961e94e1a9332f7e1805c20f9: Generated new TSK 0
I20260806 02:57:42.815667 25841 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:57:42.821398 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 25712
W20260806 02:57:42.822636 25806 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.236:34521
W20260806 02:57:42.823341 26446 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 25712: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 02:57:42.823505 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 25712
2026-08-06T02:57:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2139 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984707115702-26446-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:57:42 dist-test-slave-zj8q krb5kdc[25863](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:57:42 dist-test-slave-zj8q krb5kdc[25863](info): set up 2 sockets
Aug 06 02:57:42 dist-test-slave-zj8q krb5kdc[25863](info): commencing operation
krb5kdc: starting...
W20260806 02:57:44.068440 26446 mini_kdc.cc:121] Time spent starting KDC: real 1.199s user 0.003s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Aug 06 02:57:44 dist-test-slave-zj8q krb5kdc[25863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:57:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:57:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984707115702-26446-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984707115702-26446-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:57:44.413496 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984707115702-26446-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984707115702-26446-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984707115702-26446-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41181
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984707115702-26446-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:57:44.768436 25889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:57:44.768760 25889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:57:44.768894 25889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:57:44.776407 25889 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:57:44.776569 25889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:57:44.776667 25889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:57:44.776750 25889 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:57:44.776810 25889 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:57:44.787246 25889 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984707115702-26446-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984707115702-26446-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41181
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984707115702-26446-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984707115702-26446-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:57:44.788631 25889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:57:44.790000 25889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:57:44.802783 25900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:44.802706 25904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:44.802753 25901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:57:44.804049 25889 server_base.cc:1034] running on GCE node
I20260806 02:57:44.804958 25889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:57:44.806578 25889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:57:44.807814 25889 hybrid_clock.cc:648] HybridClock initialized: now 1785985064807758 us; error 51 us; skew 500 ppm
I20260806 02:57:44.808211 25889 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:57:44 dist-test-slave-zj8q krb5kdc[25863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985064, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:57:44.811770 25889 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:57:44.813482 25889 webserver.cc:460] Webserver started at http://127.0.0.1:35489/ using document root <none> and password file <none>
I20260806 02:57:44.814270 25889 fs_manager.cc:359] Metadata directory not provided
I20260806 02:57:44.814410 25889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:57:44.814735 25889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:57:44.817981 25889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984707115702-26446-0/minicluster-data/master-0/data/instance:
uuid: "5898934de9a7413880d785de8e7b4881"
format_stamp: "Formatted at 2026-08-06 02:57:44 on dist-test-slave-zj8q"
server_key: "33d47864b1b5d0371ea38f9600870811"
server_key_iv: "892cf7b7b3920f27c85a8821531c7b4c"
server_key_version: "encryptionkey@0"
I20260806 02:57:44.818892 25889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984707115702-26446-0/minicluster-data/master-0/wal/instance:
uuid: "5898934de9a7413880d785de8e7b4881"
format_stamp: "Formatted at 2026-08-06 02:57:44 on dist-test-slave-zj8q"
server_key: "33d47864b1b5d0371ea38f9600870811"
server_key_iv: "892cf7b7b3920f27c85a8821531c7b4c"
server_key_version: "encryptionkey@0"
I20260806 02:57:44.824749 25889 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 02:57:44.829080 25911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:44.830722 25889 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 02:57:44.830931 25889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984707115702-26446-0/minicluster-data/master-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984707115702-26446-0/minicluster-data/master-0/wal
uuid: "5898934de9a7413880d785de8e7b4881"
format_stamp: "Formatted at 2026-08-06 02:57:44 on dist-test-slave-zj8q"
server_key: "33d47864b1b5d0371ea38f9600870811"
server_key_iv: "892cf7b7b3920f27c85a8821531c7b4c"
server_key_version: "encryptionkey@0"
I20260806 02:57:44.831153 25889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984707115702-26446-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984707115702-26446-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984707115702-26446-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:57:44.848832 25889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:57:44.853156 25889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:57:44.853434 25889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:57:44.872390 25889 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41181
I20260806 02:57:44.872433 25981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41181 every 8 connection(s)
I20260806 02:57:44.873862 25889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
I20260806 02:57:44.877756 25982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:57:44.878314 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 25889
I20260806 02:57:44.878516 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1785984707115702-26446-0/minicluster-data/master-0/wal/instance
I20260806 02:57:44.879115 26446 external_mini_cluster.cc:1428] Setting key 19fe524e9b9ffa1d3489a5bc2aad223b
I20260806 02:57:44.892033 25982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5898934de9a7413880d785de8e7b4881: Bootstrap starting.
Aug 06 02:57:44 dist-test-slave-zj8q krb5kdc[25863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 02:57:44.895907 25982 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5898934de9a7413880d785de8e7b4881: Neither blocks nor log segments found. Creating new log.
I20260806 02:57:44.897111 25982 log.cc:824] T 00000000000000000000000000000000 P 5898934de9a7413880d785de8e7b4881: Log is configured to *not* fsync() on all Append() calls
I20260806 02:57:44.900792 25982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5898934de9a7413880d785de8e7b4881: No bootstrap required, opened a new log
I20260806 02:57:44.904390 25989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:44.880991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.236:59473 (local address 127.0.0.1:41181)
0806 02:57:44.881849 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:44.881862 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:44.881905 (+ 43us) server_negotiation.cc:407] Connection header received
0806 02:57:44.884299 (+ 2394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:44.884332 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:44.884832 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:44.885211 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:44.887227 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:44.888623 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:44.890059 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:44.890858 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:44.891034 (+ 176us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:44.895910 (+ 4876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:44.895920 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:44.895936 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:57:44.895989 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:44.899548 (+ 3559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:44.900666 (+ 1118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:44.900673 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:44.900680 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:44.900896 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:44.901390 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:44.901394 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:44.901396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:44.901962 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:44.902348 (+ 386us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:44.902924 (+ 576us) server_negotiation.cc:299] Negotiation successful
0806 02:57:44.903267 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":397,"threads_started":1}
I20260806 02:57:44.908257 26446 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:57:44.909056 25982 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5898934de9a7413880d785de8e7b4881 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5898934de9a7413880d785de8e7b4881" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41181 } }
I20260806 02:57:44.909415 25982 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5898934de9a7413880d785de8e7b4881 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:57:44.909505 25982 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5898934de9a7413880d785de8e7b4881 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5898934de9a7413880d785de8e7b4881, State: Initialized, Role: FOLLOWER
I20260806 02:57:44.910082 25982 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5898934de9a7413880d785de8e7b4881 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5898934de9a7413880d785de8e7b4881" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41181 } }
I20260806 02:57:44.910357 25982 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5898934de9a7413880d785de8e7b4881 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:57:44.910485 25982 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5898934de9a7413880d785de8e7b4881 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:57:44.910641 25982 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5898934de9a7413880d785de8e7b4881 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:57:44.913530 25982 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5898934de9a7413880d785de8e7b4881 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5898934de9a7413880d785de8e7b4881" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41181 } }
I20260806 02:57:44.914141 25982 leader_election.cc:302] T 00000000000000000000000000000000 P 5898934de9a7413880d785de8e7b4881 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5898934de9a7413880d785de8e7b4881; no voters:
I20260806 02:57:44.914860 25982 leader_election.cc:288] T 00000000000000000000000000000000 P 5898934de9a7413880d785de8e7b4881 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:57:44.914937 26005 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5898934de9a7413880d785de8e7b4881 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:57:44.916208 26005 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5898934de9a7413880d785de8e7b4881 [term 1 LEADER]: Becoming Leader. State: Replica: 5898934de9a7413880d785de8e7b4881, State: Running, Role: LEADER
I20260806 02:57:44.916779 26005 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5898934de9a7413880d785de8e7b4881 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5898934de9a7413880d785de8e7b4881" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41181 } }
I20260806 02:57:44.917294 25982 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5898934de9a7413880d785de8e7b4881 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:57:44.919571 26009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5898934de9a7413880d785de8e7b4881 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5898934de9a7413880d785de8e7b4881" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5898934de9a7413880d785de8e7b4881" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41181 } } }
I20260806 02:57:44.919947 26009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5898934de9a7413880d785de8e7b4881 [sys.catalog]: This master's current role is: LEADER
I20260806 02:57:44.924356 26010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5898934de9a7413880d785de8e7b4881 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5898934de9a7413880d785de8e7b4881. Latest consensus state: current_term: 1 leader_uuid: "5898934de9a7413880d785de8e7b4881" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5898934de9a7413880d785de8e7b4881" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41181 } } }
I20260806 02:57:44.925338 26010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5898934de9a7413880d785de8e7b4881 [sys.catalog]: This master's current role is: LEADER
I20260806 02:57:44.931457 25989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:44.912825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.236:35437 (local address 127.0.0.1:41181)
0806 02:57:44.912977 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:44.912984 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:44.913396 (+ 412us) server_negotiation.cc:407] Connection header received
0806 02:57:44.913571 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:44.913576 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:44.913649 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:44.913741 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:44.915874 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:44.917161 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:44.919090 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:44.919817 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:44.919919 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:44.923600 (+ 3681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:44.923607 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:44.923613 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:57:44.923656 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:44.927784 (+ 4128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:44.928795 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:44.928801 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:44.928805 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:44.928962 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:44.929466 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:44.929470 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:44.929473 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:44.930058 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:44.930415 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:44.930796 (+ 381us) server_negotiation.cc:299] Negotiation successful
0806 02:57:44.931009 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 02:57:44.931707 26013 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:57:44.937572 26013 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:57:44.947070 26013 catalog_manager.cc:1347] Generated new cluster ID: 822fa77fbb164d2c821bba17d0761f45
I20260806 02:57:44.947206 26013 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:57:44.971040 26013 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:57:44.973008 26013 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:57:44.982223 26013 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5898934de9a7413880d785de8e7b4881: Generated new TSK 0
I20260806 02:57:44.983397 26013 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:57:44.989918 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 25889
I20260806 02:57:44.991531 25975 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:57:45.084478 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 25889
W20260806 02:57:45.108072 25860 connection.cc:535] client connection to 127.0.0.1:41181 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41181): Connection reset by peer (error 104)
2026-08-06T02:57:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2257 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984707115702-26446-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:57:45 dist-test-slave-zj8q krb5kdc[26043](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:57:45 dist-test-slave-zj8q krb5kdc[26043](info): set up 2 sockets
Aug 06 02:57:45 dist-test-slave-zj8q krb5kdc[26043](info): commencing operation
krb5kdc: starting...
W20260806 02:57:46.335973 26446 mini_kdc.cc:121] Time spent starting KDC: real 1.209s user 0.001s sys 0.051s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Aug 06 02:57:46 dist-test-slave-zj8q krb5kdc[26043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:57:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:57:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984707115702-26446-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984707115702-26446-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:57:46.679845 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984707115702-26446-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984707115702-26446-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984707115702-26446-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43137
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984707115702-26446-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43137
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:57:47.033030 26066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:57:47.033365 26066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:57:47.033488 26066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:57:47.040973 26066 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:57:47.041109 26066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:57:47.041177 26066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:57:47.041266 26066 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:57:47.041318 26066 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:57:47.051328 26066 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984707115702-26446-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984707115702-26446-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43137
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984707115702-26446-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984707115702-26446-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:57:47.052685 26066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:57:47.053957 26066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:57:47.066361 26073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:47.066417 26076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:47.066445 26074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:57:47.067024 26066 server_base.cc:1034] running on GCE node
I20260806 02:57:47.068002 26066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:57:47.069430 26066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:57:47.070627 26066 hybrid_clock.cc:648] HybridClock initialized: now 1785985067070566 us; error 46 us; skew 500 ppm
I20260806 02:57:47.071053 26066 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:57:47 dist-test-slave-zj8q krb5kdc[26043](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985067, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:57:47.074649 26066 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:57:47.076162 26066 webserver.cc:460] Webserver started at http://127.0.0.1:33981/ using document root <none> and password file <none>
I20260806 02:57:47.076857 26066 fs_manager.cc:359] Metadata directory not provided
I20260806 02:57:47.076964 26066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:57:47.077242 26066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:57:47.080464 26066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984707115702-26446-0/minicluster-data/master-0/data/instance:
uuid: "4fd3aec18fc5463787622d1a0c5719b0"
format_stamp: "Formatted at 2026-08-06 02:57:47 on dist-test-slave-zj8q"
server_key: "a335843d6d34a8abfb0c249eb624e3be"
server_key_iv: "aa42b2f24cf0d0518fd8cdde067c7dea"
server_key_version: "encryptionkey@0"
I20260806 02:57:47.081265 26066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984707115702-26446-0/minicluster-data/master-0/wal/instance:
uuid: "4fd3aec18fc5463787622d1a0c5719b0"
format_stamp: "Formatted at 2026-08-06 02:57:47 on dist-test-slave-zj8q"
server_key: "a335843d6d34a8abfb0c249eb624e3be"
server_key_iv: "aa42b2f24cf0d0518fd8cdde067c7dea"
server_key_version: "encryptionkey@0"
I20260806 02:57:47.086971 26066 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260806 02:57:47.090620 26087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:47.092023 26066 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260806 02:57:47.092202 26066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984707115702-26446-0/minicluster-data/master-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984707115702-26446-0/minicluster-data/master-0/wal
uuid: "4fd3aec18fc5463787622d1a0c5719b0"
format_stamp: "Formatted at 2026-08-06 02:57:47 on dist-test-slave-zj8q"
server_key: "a335843d6d34a8abfb0c249eb624e3be"
server_key_iv: "aa42b2f24cf0d0518fd8cdde067c7dea"
server_key_version: "encryptionkey@0"
I20260806 02:57:47.092404 26066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984707115702-26446-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984707115702-26446-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984707115702-26446-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:57:47.111316 26066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:57:47.115868 26066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:57:47.116159 26066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:57:47.134491 26066 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43137
I20260806 02:57:47.134527 26163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43137 every 8 connection(s)
I20260806 02:57:47.136030 26066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
I20260806 02:57:47.140074 26164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:57:47.144750 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 26066
I20260806 02:57:47.144925 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1785984707115702-26446-0/minicluster-data/master-0/wal/instance
I20260806 02:57:47.145391 26446 external_mini_cluster.cc:1428] Setting key 891fae17471e8281d1260eb49c0ec994
I20260806 02:57:47.151669 26164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fd3aec18fc5463787622d1a0c5719b0: Bootstrap starting.
I20260806 02:57:47.155575 26164 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fd3aec18fc5463787622d1a0c5719b0: Neither blocks nor log segments found. Creating new log.
I20260806 02:57:47.156970 26164 log.cc:824] T 00000000000000000000000000000000 P 4fd3aec18fc5463787622d1a0c5719b0: Log is configured to *not* fsync() on all Append() calls
I20260806 02:57:47.160151 26168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:47.147317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.236:51921 (local address 127.0.0.1:43137)
0806 02:57:47.148117 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:47.148129 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:47.148173 (+ 44us) server_negotiation.cc:407] Connection header received
0806 02:57:47.149963 (+ 1790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:47.149989 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:47.150472 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:47.150818 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:47.151542 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:47.152898 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:47.154850 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:47.155542 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:47.155693 (+ 151us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:47.157112 (+ 1419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:47.157138 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:47.157163 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:57:47.157901 (+ 738us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:47.158011 (+ 110us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:57:47.158048 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:47.158374 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:47.158600 (+ 226us) server_negotiation.cc:299] Negotiation successful
0806 02:57:47.158817 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":366,"threads_started":1}
I20260806 02:57:47.160813 26164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fd3aec18fc5463787622d1a0c5719b0: No bootstrap required, opened a new log
I20260806 02:57:47.164608 26446 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:57:47.171314 26164 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fd3aec18fc5463787622d1a0c5719b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fd3aec18fc5463787622d1a0c5719b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43137 } }
I20260806 02:57:47.171840 26164 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4fd3aec18fc5463787622d1a0c5719b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:57:47.171968 26164 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fd3aec18fc5463787622d1a0c5719b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fd3aec18fc5463787622d1a0c5719b0, State: Initialized, Role: FOLLOWER
I20260806 02:57:47.172698 26164 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fd3aec18fc5463787622d1a0c5719b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fd3aec18fc5463787622d1a0c5719b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43137 } }
I20260806 02:57:47.173007 26164 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4fd3aec18fc5463787622d1a0c5719b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:57:47.173141 26164 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fd3aec18fc5463787622d1a0c5719b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:57:47.173316 26164 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fd3aec18fc5463787622d1a0c5719b0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:57:47.176303 26164 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fd3aec18fc5463787622d1a0c5719b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fd3aec18fc5463787622d1a0c5719b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43137 } }
I20260806 02:57:47.177052 26168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:47.170298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.236:37125 (local address 127.0.0.1:43137)
0806 02:57:47.170487 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:47.170493 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:47.170812 (+ 319us) server_negotiation.cc:407] Connection header received
0806 02:57:47.170975 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:47.170981 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:47.171067 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:47.171187 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:47.171658 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:47.172779 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:47.174272 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:47.174990 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:47.175120 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:47.176253 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:47.176259 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:47.176263 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:57:47.176292 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:47.176360 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:57:47.176381 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:47.176503 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:47.176666 (+ 163us) server_negotiation.cc:299] Negotiation successful
0806 02:57:47.176735 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 02:57:47.176955 26164 leader_election.cc:302] T 00000000000000000000000000000000 P 4fd3aec18fc5463787622d1a0c5719b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4fd3aec18fc5463787622d1a0c5719b0; no voters:
I20260806 02:57:47.177748 26164 leader_election.cc:288] T 00000000000000000000000000000000 P 4fd3aec18fc5463787622d1a0c5719b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:57:47.177871 26175 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4fd3aec18fc5463787622d1a0c5719b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:57:47.179939 26164 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fd3aec18fc5463787622d1a0c5719b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:57:47.182510 26175 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4fd3aec18fc5463787622d1a0c5719b0 [term 1 LEADER]: Becoming Leader. State: Replica: 4fd3aec18fc5463787622d1a0c5719b0, State: Running, Role: LEADER
I20260806 02:57:47.183126 26175 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4fd3aec18fc5463787622d1a0c5719b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fd3aec18fc5463787622d1a0c5719b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43137 } }
I20260806 02:57:47.186111 26179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fd3aec18fc5463787622d1a0c5719b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4fd3aec18fc5463787622d1a0c5719b0. Latest consensus state: current_term: 1 leader_uuid: "4fd3aec18fc5463787622d1a0c5719b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fd3aec18fc5463787622d1a0c5719b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43137 } } }
I20260806 02:57:47.186543 26179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fd3aec18fc5463787622d1a0c5719b0 [sys.catalog]: This master's current role is: LEADER
I20260806 02:57:47.187023 26176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fd3aec18fc5463787622d1a0c5719b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4fd3aec18fc5463787622d1a0c5719b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4fd3aec18fc5463787622d1a0c5719b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43137 } } }
I20260806 02:57:47.187335 26176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fd3aec18fc5463787622d1a0c5719b0 [sys.catalog]: This master's current role is: LEADER
W20260806 02:57:47.195560 26196 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4fd3aec18fc5463787622d1a0c5719b0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:57:47.195731 26196 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:57:47.195871 26197 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:57:47.198028 26197 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:57:47.207414 26197 catalog_manager.cc:1347] Generated new cluster ID: 4da1f19fc8dc462fa8efeacea4dc0d72
I20260806 02:57:47.207551 26197 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:57:47.228637 26197 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:57:47.230438 26197 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:57:47.241662 26197 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4fd3aec18fc5463787622d1a0c5719b0: Generated new TSK 0
I20260806 02:57:47.242887 26197 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:57:47.255582 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 26066
W20260806 02:57:47.256958 26154 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.236:51921
W20260806 02:57:47.257830 26446 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 26066: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 02:57:47.257980 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 26066
W20260806 02:57:47.280871 26039 connection.cc:535] client connection to 127.0.0.1:43137 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43137): Connection reset by peer (error 104)
2026-08-06T02:57:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2172 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984707115702-26446-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:57:47 dist-test-slave-zj8q krb5kdc[26207](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:57:47 dist-test-slave-zj8q krb5kdc[26207](info): set up 2 sockets
Aug 06 02:57:47 dist-test-slave-zj8q krb5kdc[26207](info): commencing operation
krb5kdc: starting...
W20260806 02:57:48.503894 26446 mini_kdc.cc:121] Time spent starting KDC: real 1.204s user 0.004s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Aug 06 02:57:48 dist-test-slave-zj8q krb5kdc[26207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:57:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:57:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984707115702-26446-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984707115702-26446-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 02:57:48.849820 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984707115702-26446-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984707115702-26446-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984707115702-26446-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43603
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984707115702-26446-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43603
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:57:49.202440 26227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:57:49.202752 26227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:57:49.202845 26227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:57:49.210317 26227 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:57:49.210440 26227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:57:49.210486 26227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:57:49.210554 26227 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:57:49.210597 26227 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:57:49.220425 26227 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984707115702-26446-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984707115702-26446-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43603
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984707115702-26446-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984707115702-26446-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:57:49.221670 26227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:57:49.222935 26227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:57:49.234469 26240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:49.234442 26237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:49.234571 26236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:57:49.235616 26227 server_base.cc:1034] running on GCE node
I20260806 02:57:49.236450 26227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:57:49.237972 26227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:57:49.239212 26227 hybrid_clock.cc:648] HybridClock initialized: now 1785985069239151 us; error 54 us; skew 500 ppm
I20260806 02:57:49.239578 26227 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:57:49 dist-test-slave-zj8q krb5kdc[26207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985069, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:57:49.246626 26227 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 02:57:49.248194 26227 webserver.cc:460] Webserver started at http://127.0.0.1:35525/ using document root <none> and password file <none>
I20260806 02:57:49.248884 26227 fs_manager.cc:359] Metadata directory not provided
I20260806 02:57:49.248996 26227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:57:49.249260 26227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:57:49.252329 26227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984707115702-26446-0/minicluster-data/master-0/data/instance:
uuid: "06e82ea8f9ce4442873dc051d1928d34"
format_stamp: "Formatted at 2026-08-06 02:57:49 on dist-test-slave-zj8q"
server_key: "d20d4ff0ff281610fefef2532843e377"
server_key_iv: "daab13f31ef886098718eb32bd093251"
server_key_version: "encryptionkey@0"
I20260806 02:57:49.253130 26227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984707115702-26446-0/minicluster-data/master-0/wal/instance:
uuid: "06e82ea8f9ce4442873dc051d1928d34"
format_stamp: "Formatted at 2026-08-06 02:57:49 on dist-test-slave-zj8q"
server_key: "d20d4ff0ff281610fefef2532843e377"
server_key_iv: "daab13f31ef886098718eb32bd093251"
server_key_version: "encryptionkey@0"
I20260806 02:57:49.258536 26227 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260806 02:57:49.262105 26248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:49.263545 26227 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:57:49.263705 26227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984707115702-26446-0/minicluster-data/master-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984707115702-26446-0/minicluster-data/master-0/wal
uuid: "06e82ea8f9ce4442873dc051d1928d34"
format_stamp: "Formatted at 2026-08-06 02:57:49 on dist-test-slave-zj8q"
server_key: "d20d4ff0ff281610fefef2532843e377"
server_key_iv: "daab13f31ef886098718eb32bd093251"
server_key_version: "encryptionkey@0"
I20260806 02:57:49.263887 26227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984707115702-26446-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984707115702-26446-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984707115702-26446-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:57:49.269724 26227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:57:49.274883 26227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:57:49.275223 26227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:57:49.295859 26227 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43603
I20260806 02:57:49.295899 26326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43603 every 8 connection(s)
I20260806 02:57:49.297276 26227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
I20260806 02:57:49.301023 26327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:57:49.303758 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 26227
I20260806 02:57:49.303964 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1785984707115702-26446-0/minicluster-data/master-0/wal/instance
I20260806 02:57:49.304503 26446 external_mini_cluster.cc:1428] Setting key f82765dad5023c3ad4d4d8790269c95d
I20260806 02:57:49.312664 26332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:49.306536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.236:45193 (local address 127.0.0.1:43603)
0806 02:57:49.307373 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:49.307389 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:49.307433 (+ 44us) server_negotiation.cc:407] Connection header received
0806 02:57:49.309164 (+ 1731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:49.309189 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:49.309559 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:49.309939 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:49.310962 (+ 1023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:49.310968 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:49.310985 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:57:49.311008 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:49.311068 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:57:49.311089 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:49.311406 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:49.311554 (+ 148us) server_negotiation.cc:299] Negotiation successful
0806 02:57:49.311740 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":356,"threads_started":1}
I20260806 02:57:49.314059 26327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06e82ea8f9ce4442873dc051d1928d34: Bootstrap starting.
I20260806 02:57:49.316756 26446 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:57:49.317704 26327 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06e82ea8f9ce4442873dc051d1928d34: Neither blocks nor log segments found. Creating new log.
I20260806 02:57:49.318787 26327 log.cc:824] T 00000000000000000000000000000000 P 06e82ea8f9ce4442873dc051d1928d34: Log is configured to *not* fsync() on all Append() calls
I20260806 02:57:49.321609 26327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06e82ea8f9ce4442873dc051d1928d34: No bootstrap required, opened a new log
I20260806 02:57:49.323611 26332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:49.321119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.236:49765 (local address 127.0.0.1:43603)
0806 02:57:49.321300 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:49.321307 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:49.321714 (+ 407us) server_negotiation.cc:407] Connection header received
0806 02:57:49.321895 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:49.321900 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:49.321959 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:49.322110 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:49.322962 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:49.322968 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:49.322972 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:57:49.322991 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:49.323040 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:57:49.323072 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:49.323189 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:49.323250 (+ 61us) server_negotiation.cc:299] Negotiation successful
0806 02:57:49.323311 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 02:57:49.329913 26327 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06e82ea8f9ce4442873dc051d1928d34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06e82ea8f9ce4442873dc051d1928d34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43603 } }
I20260806 02:57:49.330351 26327 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06e82ea8f9ce4442873dc051d1928d34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:57:49.330436 26327 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06e82ea8f9ce4442873dc051d1928d34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06e82ea8f9ce4442873dc051d1928d34, State: Initialized, Role: FOLLOWER
I20260806 02:57:49.330968 26327 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06e82ea8f9ce4442873dc051d1928d34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06e82ea8f9ce4442873dc051d1928d34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43603 } }
I20260806 02:57:49.331185 26327 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06e82ea8f9ce4442873dc051d1928d34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:57:49.331290 26327 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06e82ea8f9ce4442873dc051d1928d34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:57:49.331420 26327 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06e82ea8f9ce4442873dc051d1928d34 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:57:49.333889 26327 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06e82ea8f9ce4442873dc051d1928d34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06e82ea8f9ce4442873dc051d1928d34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43603 } }
I20260806 02:57:49.334391 26327 leader_election.cc:302] T 00000000000000000000000000000000 P 06e82ea8f9ce4442873dc051d1928d34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06e82ea8f9ce4442873dc051d1928d34; no voters:
I20260806 02:57:49.334942 26327 leader_election.cc:288] T 00000000000000000000000000000000 P 06e82ea8f9ce4442873dc051d1928d34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:57:49.335088 26344 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06e82ea8f9ce4442873dc051d1928d34 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:57:49.335706 26344 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06e82ea8f9ce4442873dc051d1928d34 [term 1 LEADER]: Becoming Leader. State: Replica: 06e82ea8f9ce4442873dc051d1928d34, State: Running, Role: LEADER
I20260806 02:57:49.336282 26344 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06e82ea8f9ce4442873dc051d1928d34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06e82ea8f9ce4442873dc051d1928d34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43603 } }
I20260806 02:57:49.336930 26327 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06e82ea8f9ce4442873dc051d1928d34 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:57:49.339325 26347 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06e82ea8f9ce4442873dc051d1928d34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06e82ea8f9ce4442873dc051d1928d34. Latest consensus state: current_term: 1 leader_uuid: "06e82ea8f9ce4442873dc051d1928d34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06e82ea8f9ce4442873dc051d1928d34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43603 } } }
I20260806 02:57:49.339746 26347 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06e82ea8f9ce4442873dc051d1928d34 [sys.catalog]: This master's current role is: LEADER
I20260806 02:57:49.340384 26346 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06e82ea8f9ce4442873dc051d1928d34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06e82ea8f9ce4442873dc051d1928d34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06e82ea8f9ce4442873dc051d1928d34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43603 } } }
I20260806 02:57:49.340651 26353 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:57:49.340708 26346 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06e82ea8f9ce4442873dc051d1928d34 [sys.catalog]: This master's current role is: LEADER
I20260806 02:57:49.346127 26353 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:57:49.356986 26353 catalog_manager.cc:1347] Generated new cluster ID: f5379ab4fdfb4d61ae235c9021ae5681
I20260806 02:57:49.357115 26353 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:57:49.372910 26353 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:57:49.373440 26353 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:57:49.388696 26353 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06e82ea8f9ce4442873dc051d1928d34: Generated new TSK 0
I20260806 02:57:49.389829 26353 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:57:49.406759 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 26227
W20260806 02:57:49.407820 26322 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.236:45193
W20260806 02:57:49.408488 26446 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 26227: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 02:57:49.408646 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 26227
2026-08-06T02:57:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2147 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984707115702-26446-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:57:49 dist-test-slave-zj8q krb5kdc[26386](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:57:49 dist-test-slave-zj8q krb5kdc[26386](info): set up 2 sockets
Aug 06 02:57:49 dist-test-slave-zj8q krb5kdc[26386](info): commencing operation
krb5kdc: starting...
W20260806 02:57:50.650934 26446 mini_kdc.cc:121] Time spent starting KDC: real 1.203s user 0.004s sys 0.049s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Aug 06 02:57:50 dist-test-slave-zj8q krb5kdc[26386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:57:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:57:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.211.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab.
Entry for principal kudu/127.25.211.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab.
I20260806 02:57:50.997649 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984707115702-26446-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984707115702-26446-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984707115702-26446-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:35893
--webserver_interface=127.25.211.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:39869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.211.190:35893
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:57:51.356770 26418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:57:51.357205 26418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:57:51.357365 26418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:57:51.364907 26418 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:57:51.365069 26418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:57:51.365152 26418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:57:51.365242 26418 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:57:51.365309 26418 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:57:51.375785 26418 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:39869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984707115702-26446-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984707115702-26446-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.211.190:35893
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:35893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.211.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984707115702-26446-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:57:51.377233 26418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:57:51.378715 26418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:57:51.391213 26426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:51.391140 26432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:51.391247 26427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:57:51.392091 26418 server_base.cc:1034] running on GCE node
I20260806 02:57:51.392875 26418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:57:51.394351 26418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:57:51.395569 26418 hybrid_clock.cc:648] HybridClock initialized: now 1785985071395512 us; error 57 us; skew 500 ppm
I20260806 02:57:51.395978 26418 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:57:51 dist-test-slave-zj8q krb5kdc[26386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985071, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:57:51.399245 26418 init.cc:377] Logged in from keytab as kudu/127.25.211.190@KRBTEST.COM (short username kudu)
I20260806 02:57:51.400897 26418 webserver.cc:460] Webserver started at http://127.25.211.190:36415/ using document root <none> and password file <none>
I20260806 02:57:51.401674 26418 fs_manager.cc:359] Metadata directory not provided
I20260806 02:57:51.401790 26418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:57:51.402063 26418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:57:51.405333 26418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984707115702-26446-0/minicluster-data/master-0/data/instance:
uuid: "c325892057a04613884e90c8366153e7"
format_stamp: "Formatted at 2026-08-06 02:57:51 on dist-test-slave-zj8q"
server_key: "bf3c4ffb23f58a6d78bd515e97281740"
server_key_iv: "8d29a19b794a6e54e71c3fc9629b66a9"
server_key_version: "encryptionkey@0"
I20260806 02:57:51.406214 26418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984707115702-26446-0/minicluster-data/master-0/wal/instance:
uuid: "c325892057a04613884e90c8366153e7"
format_stamp: "Formatted at 2026-08-06 02:57:51 on dist-test-slave-zj8q"
server_key: "bf3c4ffb23f58a6d78bd515e97281740"
server_key_iv: "8d29a19b794a6e54e71c3fc9629b66a9"
server_key_version: "encryptionkey@0"
I20260806 02:57:51.412119 26418 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260806 02:57:51.415812 26454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:51.417272 26418 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260806 02:57:51.417464 26418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984707115702-26446-0/minicluster-data/master-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984707115702-26446-0/minicluster-data/master-0/wal
uuid: "c325892057a04613884e90c8366153e7"
format_stamp: "Formatted at 2026-08-06 02:57:51 on dist-test-slave-zj8q"
server_key: "bf3c4ffb23f58a6d78bd515e97281740"
server_key_iv: "8d29a19b794a6e54e71c3fc9629b66a9"
server_key_version: "encryptionkey@0"
I20260806 02:57:51.417659 26418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984707115702-26446-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984707115702-26446-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984707115702-26446-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:57:51.441629 26418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:57:51.446239 26418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:57:51.446547 26418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:57:51.465000 26418 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.190:35893
I20260806 02:57:51.465042 26545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.190:35893 every 8 connection(s)
I20260806 02:57:51.466523 26418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
I20260806 02:57:51.470512 26547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:57:51.472766 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 26418
I20260806 02:57:51.472946 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1785984707115702-26446-0/minicluster-data/master-0/wal/instance
I20260806 02:57:51.473500 26446 external_mini_cluster.cc:1428] Setting key 951665d109dfa04752977b74bd023d6a
I20260806 02:57:51.483878 26547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c325892057a04613884e90c8366153e7: Bootstrap starting.
I20260806 02:57:51.488055 26547 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c325892057a04613884e90c8366153e7: Neither blocks nor log segments found. Creating new log.
Aug 06 02:57:51 dist-test-slave-zj8q krb5kdc[26386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:57:51.489282 26547 log.cc:824] T 00000000000000000000000000000000 P c325892057a04613884e90c8366153e7: Log is configured to *not* fsync() on all Append() calls
I20260806 02:57:51.492214 26547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c325892057a04613884e90c8366153e7: No bootstrap required, opened a new log
I20260806 02:57:51.497871 26552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:51.475502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37239 (local address 127.25.211.190:35893)
0806 02:57:51.476424 (+ 922us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:51.476440 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:51.476492 (+ 52us) server_negotiation.cc:407] Connection header received
0806 02:57:51.478361 (+ 1869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:51.478391 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:51.478874 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:51.479244 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:51.481275 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:51.482814 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:51.484604 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:51.485349 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:51.485509 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:51.490342 (+ 4833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:51.490352 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:51.490371 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:57:51.490412 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:51.493382 (+ 2970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:51.494240 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:51.494247 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:51.494254 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:51.494442 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:51.494935 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:51.494939 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:51.494941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:51.495539 (+ 598us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:51.495871 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:51.496422 (+ 551us) server_negotiation.cc:299] Negotiation successful
0806 02:57:51.496770 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":433,"threads_started":1}
I20260806 02:57:51.500278 26547 raft_consensus.cc:348] T 00000000000000000000000000000000 P c325892057a04613884e90c8366153e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c325892057a04613884e90c8366153e7" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 35893 } }
I20260806 02:57:51.500730 26547 raft_consensus.cc:374] T 00000000000000000000000000000000 P c325892057a04613884e90c8366153e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:57:51.500824 26547 raft_consensus.cc:729] T 00000000000000000000000000000000 P c325892057a04613884e90c8366153e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c325892057a04613884e90c8366153e7, State: Initialized, Role: FOLLOWER
I20260806 02:57:51.501430 26547 consensus_queue.cc:260] T 00000000000000000000000000000000 P c325892057a04613884e90c8366153e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c325892057a04613884e90c8366153e7" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 35893 } }
I20260806 02:57:51.501682 26547 raft_consensus.cc:388] T 00000000000000000000000000000000 P c325892057a04613884e90c8366153e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:57:51.501825 26547 raft_consensus.cc:482] T 00000000000000000000000000000000 P c325892057a04613884e90c8366153e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:57:51.501994 26446 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:57:51.502030 26547 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c325892057a04613884e90c8366153e7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:57:51.504314 26547 raft_consensus.cc:504] T 00000000000000000000000000000000 P c325892057a04613884e90c8366153e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c325892057a04613884e90c8366153e7" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 35893 } }
I20260806 02:57:51.504870 26547 leader_election.cc:302] T 00000000000000000000000000000000 P c325892057a04613884e90c8366153e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c325892057a04613884e90c8366153e7; no voters:
I20260806 02:57:51.505606 26547 leader_election.cc:288] T 00000000000000000000000000000000 P c325892057a04613884e90c8366153e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:57:51.505726 26562 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c325892057a04613884e90c8366153e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:57:51.507576 26547 sys_catalog.cc:564] T 00000000000000000000000000000000 P c325892057a04613884e90c8366153e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:57:51.510447 26562 raft_consensus.cc:686] T 00000000000000000000000000000000 P c325892057a04613884e90c8366153e7 [term 1 LEADER]: Becoming Leader. State: Replica: c325892057a04613884e90c8366153e7, State: Running, Role: LEADER
I20260806 02:57:51.511234 26562 consensus_queue.cc:237] T 00000000000000000000000000000000 P c325892057a04613884e90c8366153e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c325892057a04613884e90c8366153e7" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 35893 } }
I20260806 02:57:51.514619 26571 sys_catalog.cc:455] T 00000000000000000000000000000000 P c325892057a04613884e90c8366153e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c325892057a04613884e90c8366153e7. Latest consensus state: current_term: 1 leader_uuid: "c325892057a04613884e90c8366153e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c325892057a04613884e90c8366153e7" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 35893 } } }
I20260806 02:57:51.515094 26571 sys_catalog.cc:458] T 00000000000000000000000000000000 P c325892057a04613884e90c8366153e7 [sys.catalog]: This master's current role is: LEADER
I20260806 02:57:51.514724 26563 sys_catalog.cc:455] T 00000000000000000000000000000000 P c325892057a04613884e90c8366153e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c325892057a04613884e90c8366153e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c325892057a04613884e90c8366153e7" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 35893 } } }
I20260806 02:57:51.516573 26563 sys_catalog.cc:458] T 00000000000000000000000000000000 P c325892057a04613884e90c8366153e7 [sys.catalog]: This master's current role is: LEADER
I20260806 02:57:51.527395 26552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:51.506522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47265 (local address 127.25.211.190:35893)
0806 02:57:51.506710 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:51.506716 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:51.510099 (+ 3383us) server_negotiation.cc:407] Connection header received
0806 02:57:51.510354 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:51.510362 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:51.510464 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:51.510574 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:51.512917 (+ 2343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:51.514117 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:51.516181 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:51.516934 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:51.517072 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:51.520657 (+ 3585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:51.520665 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:51.520670 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:57:51.520719 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:51.524293 (+ 3574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:51.525082 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:51.525085 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:51.525088 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:51.525208 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:51.525686 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:51.525690 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:51.525692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:51.526002 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:51.526283 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:51.526705 (+ 422us) server_negotiation.cc:299] Negotiation successful
0806 02:57:51.526894 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260806 02:57:51.529158 26583 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c325892057a04613884e90c8366153e7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:57:51.529301 26583 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:57:51.529418 26586 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:57:51.531749 26586 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:57:51.541064 26586 catalog_manager.cc:1347] Generated new cluster ID: 42ca981e061b4ddc9aac2571fa85d149
I20260806 02:57:51.541193 26586 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:57:51.568104 26586 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:57:51.569516 26586 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:57:51.578711 26586 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c325892057a04613884e90c8366153e7: Generated new TSK 0
I20260806 02:57:51.579608 26586 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:57:51.586450 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 26418
I20260806 02:57:51.587664 26536 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:57:51.681419 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 26418
W20260806 02:57:51.709414 26381 connection.cc:535] client connection to 127.25.211.190:35893 recv error: Network error: failed to read from TLS socket (remote: 127.25.211.190:35893): Connection reset by peer (error 104)
2026-08-06T02:57:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2280 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984707115702-26446-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:57:51 dist-test-slave-zj8q krb5kdc[26598](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:57:51 dist-test-slave-zj8q krb5kdc[26598](info): set up 2 sockets
Aug 06 02:57:51 dist-test-slave-zj8q krb5kdc[26598](info): commencing operation
krb5kdc: starting...
W20260806 02:57:52.938290 26446 mini_kdc.cc:121] Time spent starting KDC: real 1.210s user 0.000s sys 0.056s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Aug 06 02:57:53 dist-test-slave-zj8q krb5kdc[26598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:57:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:57:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.211.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab.
Entry for principal kudu/127.25.211.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab.
I20260806 02:57:53.287355 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984707115702-26446-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984707115702-26446-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984707115702-26446-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:37901
--webserver_interface=127.25.211.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:37657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.211.190:37901
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:57:53.645325 26616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:57:53.645692 26616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:57:53.645825 26616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:57:53.653364 26616 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:57:53.653515 26616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:57:53.653594 26616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:57:53.653677 26616 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:57:53.653738 26616 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:57:53.664163 26616 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:37657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984707115702-26446-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984707115702-26446-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.211.190:37901
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:37901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.211.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984707115702-26446-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:57:53.665596 26616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:57:53.667006 26616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:57:53.679087 26624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:53.679134 26625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:53.679087 26627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:57:53.680290 26616 server_base.cc:1034] running on GCE node
I20260806 02:57:53.681100 26616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:57:53.682559 26616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:57:53.683765 26616 hybrid_clock.cc:648] HybridClock initialized: now 1785985073683721 us; error 32 us; skew 500 ppm
I20260806 02:57:53.684115 26616 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:57:53 dist-test-slave-zj8q krb5kdc[26598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985073, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:57:53.687556 26616 init.cc:377] Logged in from keytab as kudu/127.25.211.190@KRBTEST.COM (short username kudu)
I20260806 02:57:53.689127 26616 webserver.cc:460] Webserver started at http://127.25.211.190:46055/ using document root <none> and password file <none>
I20260806 02:57:53.689848 26616 fs_manager.cc:359] Metadata directory not provided
I20260806 02:57:53.689977 26616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:57:53.690335 26616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:57:53.693464 26616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984707115702-26446-0/minicluster-data/master-0/data/instance:
uuid: "2e933979e2cc40fd8034969afe98835c"
format_stamp: "Formatted at 2026-08-06 02:57:53 on dist-test-slave-zj8q"
server_key: "b20fdc63884e687ca17750fd81c8ac39"
server_key_iv: "2330b9217da5a8a30c600d45630dc61a"
server_key_version: "encryptionkey@0"
I20260806 02:57:53.694335 26616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984707115702-26446-0/minicluster-data/master-0/wal/instance:
uuid: "2e933979e2cc40fd8034969afe98835c"
format_stamp: "Formatted at 2026-08-06 02:57:53 on dist-test-slave-zj8q"
server_key: "b20fdc63884e687ca17750fd81c8ac39"
server_key_iv: "2330b9217da5a8a30c600d45630dc61a"
server_key_version: "encryptionkey@0"
I20260806 02:57:53.699919 26616 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 02:57:53.703518 26642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:53.704828 26616 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:57:53.705016 26616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984707115702-26446-0/minicluster-data/master-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984707115702-26446-0/minicluster-data/master-0/wal
uuid: "2e933979e2cc40fd8034969afe98835c"
format_stamp: "Formatted at 2026-08-06 02:57:53 on dist-test-slave-zj8q"
server_key: "b20fdc63884e687ca17750fd81c8ac39"
server_key_iv: "2330b9217da5a8a30c600d45630dc61a"
server_key_version: "encryptionkey@0"
I20260806 02:57:53.705224 26616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984707115702-26446-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984707115702-26446-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984707115702-26446-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:57:53.722263 26616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:57:53.726745 26616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:57:53.727008 26616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:57:53.745122 26616 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.190:37901
I20260806 02:57:53.745159 26715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.190:37901 every 8 connection(s)
I20260806 02:57:53.746677 26616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
I20260806 02:57:53.750666 26716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:57:53.751928 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 26616
I20260806 02:57:53.752100 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1785984707115702-26446-0/minicluster-data/master-0/wal/instance
I20260806 02:57:53.752527 26446 external_mini_cluster.cc:1428] Setting key 9825f649a26442568b5d7ad7abe28613
I20260806 02:57:53.764200 26716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e933979e2cc40fd8034969afe98835c: Bootstrap starting.
I20260806 02:57:53.765839 26720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:53.754245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33565 (local address 127.25.211.190:37901)
0806 02:57:53.755056 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:53.755068 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:53.755111 (+ 43us) server_negotiation.cc:407] Connection header received
0806 02:57:53.756767 (+ 1656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:53.756790 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:53.757226 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:53.757563 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:53.758200 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:53.759497 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:53.761380 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:53.762106 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:53.762291 (+ 185us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:53.763503 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:53.763511 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:53.763525 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:57:53.764068 (+ 543us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:53.764163 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:57:53.764192 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:53.764312 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:53.764609 (+ 297us) server_negotiation.cc:299] Negotiation successful
0806 02:57:53.764814 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":377,"threads_started":1}
I20260806 02:57:53.767623 26716 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e933979e2cc40fd8034969afe98835c: Neither blocks nor log segments found. Creating new log.
I20260806 02:57:53.768769 26716 log.cc:824] T 00000000000000000000000000000000 P 2e933979e2cc40fd8034969afe98835c: Log is configured to *not* fsync() on all Append() calls
I20260806 02:57:53.769762 26446 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:57:53.771781 26716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e933979e2cc40fd8034969afe98835c: No bootstrap required, opened a new log
I20260806 02:57:53.780089 26716 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e933979e2cc40fd8034969afe98835c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e933979e2cc40fd8034969afe98835c" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 37901 } }
I20260806 02:57:53.780520 26720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:53.774215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54073 (local address 127.25.211.190:37901)
0806 02:57:53.774395 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:53.774406 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:53.774801 (+ 395us) server_negotiation.cc:407] Connection header received
0806 02:57:53.774972 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:53.774976 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:53.775038 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:53.775138 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:53.775667 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:53.776499 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:53.777871 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:53.778425 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:53.778551 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:53.779807 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:53.779812 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:53.779815 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:57:53.779835 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:53.779877 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:57:53.779893 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:53.779986 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:53.780198 (+ 212us) server_negotiation.cc:299] Negotiation successful
0806 02:57:53.780256 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 02:57:53.780635 26716 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e933979e2cc40fd8034969afe98835c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:57:53.780784 26716 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e933979e2cc40fd8034969afe98835c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e933979e2cc40fd8034969afe98835c, State: Initialized, Role: FOLLOWER
I20260806 02:57:53.781457 26716 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e933979e2cc40fd8034969afe98835c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e933979e2cc40fd8034969afe98835c" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 37901 } }
I20260806 02:57:53.781716 26716 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e933979e2cc40fd8034969afe98835c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:57:53.781867 26716 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e933979e2cc40fd8034969afe98835c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:57:53.782078 26716 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e933979e2cc40fd8034969afe98835c [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:57:53.784402 26716 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e933979e2cc40fd8034969afe98835c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e933979e2cc40fd8034969afe98835c" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 37901 } }
I20260806 02:57:53.785017 26716 leader_election.cc:302] T 00000000000000000000000000000000 P 2e933979e2cc40fd8034969afe98835c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e933979e2cc40fd8034969afe98835c; no voters:
I20260806 02:57:53.785727 26716 leader_election.cc:288] T 00000000000000000000000000000000 P 2e933979e2cc40fd8034969afe98835c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:57:53.785843 26732 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e933979e2cc40fd8034969afe98835c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:57:53.786516 26732 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e933979e2cc40fd8034969afe98835c [term 1 LEADER]: Becoming Leader. State: Replica: 2e933979e2cc40fd8034969afe98835c, State: Running, Role: LEADER
I20260806 02:57:53.787087 26732 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e933979e2cc40fd8034969afe98835c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e933979e2cc40fd8034969afe98835c" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 37901 } }
I20260806 02:57:53.787854 26716 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e933979e2cc40fd8034969afe98835c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:57:53.789661 26733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e933979e2cc40fd8034969afe98835c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e933979e2cc40fd8034969afe98835c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e933979e2cc40fd8034969afe98835c" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 37901 } } }
I20260806 02:57:53.790122 26733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e933979e2cc40fd8034969afe98835c [sys.catalog]: This master's current role is: LEADER
I20260806 02:57:53.789760 26735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e933979e2cc40fd8034969afe98835c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e933979e2cc40fd8034969afe98835c. Latest consensus state: current_term: 1 leader_uuid: "2e933979e2cc40fd8034969afe98835c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e933979e2cc40fd8034969afe98835c" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 37901 } } }
I20260806 02:57:53.790541 26735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e933979e2cc40fd8034969afe98835c [sys.catalog]: This master's current role is: LEADER
I20260806 02:57:53.792469 26743 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:57:53.798635 26743 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:57:53.808125 26743 catalog_manager.cc:1347] Generated new cluster ID: cdd980a0447245608d90318510c151a3
I20260806 02:57:53.808254 26743 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:57:53.824576 26743 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:57:53.825966 26743 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:57:53.838451 26743 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e933979e2cc40fd8034969afe98835c: Generated new TSK 0
I20260806 02:57:53.839689 26743 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:57:53.849279 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 26616
W20260806 02:57:53.850570 26709 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33565
W20260806 02:57:53.851296 26446 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 26616: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 02:57:53.851424 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 26616
W20260806 02:57:53.881865 26594 connection.cc:535] client connection to 127.25.211.190:37901 recv error: Network error: failed to read from TLS socket (remote: 127.25.211.190:37901): Connection reset by peer (error 104)
2026-08-06T02:57:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2172 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984707115702-26446-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:57:53 dist-test-slave-zj8q krb5kdc[26764](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:57:53 dist-test-slave-zj8q krb5kdc[26764](info): set up 2 sockets
Aug 06 02:57:53 dist-test-slave-zj8q krb5kdc[26764](info): commencing operation
krb5kdc: starting...
W20260806 02:57:55.103531 26446 mini_kdc.cc:121] Time spent starting KDC: real 1.202s user 0.000s sys 0.053s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Aug 06 02:57:55 dist-test-slave-zj8q krb5kdc[26764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:57:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:57:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.211.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab.
Entry for principal kudu/127.25.211.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab.
I20260806 02:57:55.449684 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984707115702-26446-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984707115702-26446-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984707115702-26446-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:41021
--webserver_interface=127.25.211.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:42803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.211.190:41021
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:57:55.808079 26783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:57:55.808457 26783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:57:55.808598 26783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:57:55.816268 26783 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:57:55.816448 26783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:57:55.816535 26783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:57:55.816622 26783 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:57:55.816680 26783 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:57:55.827312 26783 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:42803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984707115702-26446-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984707115702-26446-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.211.190:41021
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:41021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.211.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984707115702-26446-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:57:55.828727 26783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:57:55.830087 26783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:57:55.842043 26789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:55.841984 26790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:55.841959 26795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:57:55.843127 26783 server_base.cc:1034] running on GCE node
I20260806 02:57:55.844033 26783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:57:55.845597 26783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:57:55.846825 26783 hybrid_clock.cc:648] HybridClock initialized: now 1785985075846782 us; error 45 us; skew 500 ppm
I20260806 02:57:55.847213 26783 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
W20260806 02:57:56.354382 26783 net_util.cc:504] Time spent look up canonical hostname for localhost 'dist-test-slave-zj8q': real 0.507s user 0.000s sys 0.002s
Aug 06 02:57:56 dist-test-slave-zj8q krb5kdc[26764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985076, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:57:56.358110 26783 init.cc:377] Logged in from keytab as kudu/127.25.211.190@KRBTEST.COM (short username kudu)
I20260806 02:57:56.367463 26783 webserver.cc:460] Webserver started at http://127.25.211.190:33533/ using document root <none> and password file <none>
I20260806 02:57:56.368541 26783 fs_manager.cc:359] Metadata directory not provided
I20260806 02:57:56.368736 26783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:57:56.369194 26783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:57:56.373798 26783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984707115702-26446-0/minicluster-data/master-0/data/instance:
uuid: "e921d2535f66468796e98590b90855a1"
format_stamp: "Formatted at 2026-08-06 02:57:56 on dist-test-slave-zj8q"
server_key: "242b2db7f5c049c36214b21fd5a93a86"
server_key_iv: "f5a0bdf4449f640732ecf11780bc4bd7"
server_key_version: "encryptionkey@0"
I20260806 02:57:56.374907 26783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984707115702-26446-0/minicluster-data/master-0/wal/instance:
uuid: "e921d2535f66468796e98590b90855a1"
format_stamp: "Formatted at 2026-08-06 02:57:56 on dist-test-slave-zj8q"
server_key: "242b2db7f5c049c36214b21fd5a93a86"
server_key_iv: "f5a0bdf4449f640732ecf11780bc4bd7"
server_key_version: "encryptionkey@0"
I20260806 02:57:56.382014 26783 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260806 02:57:56.393896 26810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:56.396075 26783 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.006s sys 0.001s
I20260806 02:57:56.396282 26783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984707115702-26446-0/minicluster-data/master-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984707115702-26446-0/minicluster-data/master-0/wal
uuid: "e921d2535f66468796e98590b90855a1"
format_stamp: "Formatted at 2026-08-06 02:57:56 on dist-test-slave-zj8q"
server_key: "242b2db7f5c049c36214b21fd5a93a86"
server_key_iv: "f5a0bdf4449f640732ecf11780bc4bd7"
server_key_version: "encryptionkey@0"
I20260806 02:57:56.396508 26783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984707115702-26446-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984707115702-26446-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984707115702-26446-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:57:56.404475 26783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:57:56.410442 26783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:57:56.410821 26783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:57:56.430419 26783 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.190:41021
I20260806 02:57:56.430438 26889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.190:41021 every 8 connection(s)
I20260806 02:57:56.431866 26783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
I20260806 02:57:56.435879 26891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:57:56.437458 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 26783
I20260806 02:57:56.437645 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1785984707115702-26446-0/minicluster-data/master-0/wal/instance
I20260806 02:57:56.438220 26446 external_mini_cluster.cc:1428] Setting key 0e01079ddfea63e9483e9835ff8310ac
I20260806 02:57:56.447001 26895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:56.440306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51083 (local address 127.25.211.190:41021)
0806 02:57:56.441196 (+ 890us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:56.441209 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:56.441256 (+ 47us) server_negotiation.cc:407] Connection header received
0806 02:57:56.443037 (+ 1781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:56.443065 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:56.443494 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:56.443872 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:56.445199 (+ 1327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:56.445207 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:56.445223 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:57:56.445242 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:56.445317 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:57:56.445343 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:56.445591 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:56.445714 (+ 123us) server_negotiation.cc:299] Negotiation successful
0806 02:57:56.445934 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":400,"threads_started":1}
I20260806 02:57:56.451418 26446 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:57:56.451427 26891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e921d2535f66468796e98590b90855a1: Bootstrap starting.
I20260806 02:57:56.455760 26891 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e921d2535f66468796e98590b90855a1: Neither blocks nor log segments found. Creating new log.
I20260806 02:57:56.457243 26891 log.cc:824] T 00000000000000000000000000000000 P e921d2535f66468796e98590b90855a1: Log is configured to *not* fsync() on all Append() calls
I20260806 02:57:56.459618 26895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:56.456969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59615 (local address 127.25.211.190:41021)
0806 02:57:56.457210 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:56.457215 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:56.457548 (+ 333us) server_negotiation.cc:407] Connection header received
0806 02:57:56.457763 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:56.457767 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:56.457841 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:56.457947 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:56.458939 (+ 992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:56.458943 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:56.458946 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:57:56.458961 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:56.459001 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:57:56.459016 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:56.459110 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:56.459245 (+ 135us) server_negotiation.cc:299] Negotiation successful
0806 02:57:56.459293 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260806 02:57:56.460899 26891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e921d2535f66468796e98590b90855a1: No bootstrap required, opened a new log
I20260806 02:57:56.468541 26891 raft_consensus.cc:348] T 00000000000000000000000000000000 P e921d2535f66468796e98590b90855a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e921d2535f66468796e98590b90855a1" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 41021 } }
I20260806 02:57:56.468946 26891 raft_consensus.cc:374] T 00000000000000000000000000000000 P e921d2535f66468796e98590b90855a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:57:56.469034 26891 raft_consensus.cc:729] T 00000000000000000000000000000000 P e921d2535f66468796e98590b90855a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e921d2535f66468796e98590b90855a1, State: Initialized, Role: FOLLOWER
I20260806 02:57:56.469635 26891 consensus_queue.cc:260] T 00000000000000000000000000000000 P e921d2535f66468796e98590b90855a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e921d2535f66468796e98590b90855a1" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 41021 } }
I20260806 02:57:56.469882 26891 raft_consensus.cc:388] T 00000000000000000000000000000000 P e921d2535f66468796e98590b90855a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:57:56.470031 26891 raft_consensus.cc:482] T 00000000000000000000000000000000 P e921d2535f66468796e98590b90855a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:57:56.470260 26891 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e921d2535f66468796e98590b90855a1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:57:56.472698 26891 raft_consensus.cc:504] T 00000000000000000000000000000000 P e921d2535f66468796e98590b90855a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e921d2535f66468796e98590b90855a1" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 41021 } }
I20260806 02:57:56.473224 26891 leader_election.cc:302] T 00000000000000000000000000000000 P e921d2535f66468796e98590b90855a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e921d2535f66468796e98590b90855a1; no voters:
I20260806 02:57:56.473816 26891 leader_election.cc:288] T 00000000000000000000000000000000 P e921d2535f66468796e98590b90855a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:57:56.473970 26911 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e921d2535f66468796e98590b90855a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:57:56.474689 26911 raft_consensus.cc:686] T 00000000000000000000000000000000 P e921d2535f66468796e98590b90855a1 [term 1 LEADER]: Becoming Leader. State: Replica: e921d2535f66468796e98590b90855a1, State: Running, Role: LEADER
I20260806 02:57:56.475306 26911 consensus_queue.cc:237] T 00000000000000000000000000000000 P e921d2535f66468796e98590b90855a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e921d2535f66468796e98590b90855a1" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 41021 } }
I20260806 02:57:56.476013 26891 sys_catalog.cc:564] T 00000000000000000000000000000000 P e921d2535f66468796e98590b90855a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:57:56.478492 26912 sys_catalog.cc:455] T 00000000000000000000000000000000 P e921d2535f66468796e98590b90855a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e921d2535f66468796e98590b90855a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e921d2535f66468796e98590b90855a1" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 41021 } } }
I20260806 02:57:56.478468 26913 sys_catalog.cc:455] T 00000000000000000000000000000000 P e921d2535f66468796e98590b90855a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e921d2535f66468796e98590b90855a1. Latest consensus state: current_term: 1 leader_uuid: "e921d2535f66468796e98590b90855a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e921d2535f66468796e98590b90855a1" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 41021 } } }
I20260806 02:57:56.478950 26912 sys_catalog.cc:458] T 00000000000000000000000000000000 P e921d2535f66468796e98590b90855a1 [sys.catalog]: This master's current role is: LEADER
I20260806 02:57:56.479004 26913 sys_catalog.cc:458] T 00000000000000000000000000000000 P e921d2535f66468796e98590b90855a1 [sys.catalog]: This master's current role is: LEADER
I20260806 02:57:56.481379 26922 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:57:56.486935 26922 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:57:56.497134 26922 catalog_manager.cc:1347] Generated new cluster ID: 3bf5bf6c00684cc8980b778f9c7ff0e0
I20260806 02:57:56.497291 26922 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:57:56.515970 26922 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:57:56.516461 26922 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:57:56.525107 26922 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e921d2535f66468796e98590b90855a1: Generated new TSK 0
I20260806 02:57:56.526095 26922 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:57:56.537031 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 26783
W20260806 02:57:56.538364 26882 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51083
W20260806 02:57:56.539144 26446 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 26783: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 02:57:56.539284 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 26783
2026-08-06T02:57:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2683 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984707115702-26446-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:57:56 dist-test-slave-zj8q krb5kdc[26938](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:57:56 dist-test-slave-zj8q krb5kdc[26938](info): set up 2 sockets
Aug 06 02:57:56 dist-test-slave-zj8q krb5kdc[26938](info): commencing operation
krb5kdc: starting...
W20260806 02:57:57.784789 26446 mini_kdc.cc:121] Time spent starting KDC: real 1.200s user 0.003s sys 0.051s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Aug 06 02:57:58 dist-test-slave-zj8q krb5kdc[26938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:57:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:57:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.211.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab.
Entry for principal kudu/127.25.211.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab.
I20260806 02:57:58.139655 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984707115702-26446-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984707115702-26446-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984707115702-26446-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:45439
--webserver_interface=127.25.211.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:39195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.211.190:45439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:57:58.500193 26961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:57:58.500514 26961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:57:58.500660 26961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:57:58.508365 26961 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:57:58.508520 26961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:57:58.508607 26961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:57:58.508711 26961 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:57:58.508769 26961 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:57:58.519505 26961 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:39195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984707115702-26446-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984707115702-26446-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.211.190:45439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:45439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.211.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984707115702-26446-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:57:58.520920 26961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:57:58.522323 26961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:57:58.534467 26969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:58.534503 26971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:57:58.534551 26973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:57:58.535324 26961 server_base.cc:1034] running on GCE node
I20260806 02:57:58.536185 26961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:57:58.537725 26961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:57:58.538932 26961 hybrid_clock.cc:648] HybridClock initialized: now 1785985078538888 us; error 36 us; skew 500 ppm
I20260806 02:57:58.539304 26961 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:57:58 dist-test-slave-zj8q krb5kdc[26938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985078, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:57:58.542506 26961 init.cc:377] Logged in from keytab as kudu/127.25.211.190@KRBTEST.COM (short username kudu)
I20260806 02:57:58.544148 26961 webserver.cc:460] Webserver started at http://127.25.211.190:36485/ using document root <none> and password file <none>
I20260806 02:57:58.544943 26961 fs_manager.cc:359] Metadata directory not provided
I20260806 02:57:58.545071 26961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:57:58.545393 26961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:57:58.548667 26961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984707115702-26446-0/minicluster-data/master-0/data/instance:
uuid: "c778123ff879448b9222139c48f46175"
format_stamp: "Formatted at 2026-08-06 02:57:58 on dist-test-slave-zj8q"
server_key: "618f174b51169b4a48e90b0e51ccf91b"
server_key_iv: "7e5cbb214740d696e7d0aa83f7286adb"
server_key_version: "encryptionkey@0"
I20260806 02:57:58.549487 26961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984707115702-26446-0/minicluster-data/master-0/wal/instance:
uuid: "c778123ff879448b9222139c48f46175"
format_stamp: "Formatted at 2026-08-06 02:57:58 on dist-test-slave-zj8q"
server_key: "618f174b51169b4a48e90b0e51ccf91b"
server_key_iv: "7e5cbb214740d696e7d0aa83f7286adb"
server_key_version: "encryptionkey@0"
I20260806 02:57:58.555593 26961 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260806 02:57:58.559520 26982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:57:58.561122 26961 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:57:58.561328 26961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984707115702-26446-0/minicluster-data/master-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984707115702-26446-0/minicluster-data/master-0/wal
uuid: "c778123ff879448b9222139c48f46175"
format_stamp: "Formatted at 2026-08-06 02:57:58 on dist-test-slave-zj8q"
server_key: "618f174b51169b4a48e90b0e51ccf91b"
server_key_iv: "7e5cbb214740d696e7d0aa83f7286adb"
server_key_version: "encryptionkey@0"
I20260806 02:57:58.561543 26961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984707115702-26446-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984707115702-26446-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984707115702-26446-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:57:58.577019 26961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:57:58.581671 26961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:57:58.581951 26961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:57:58.600525 26961 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.190:45439
I20260806 02:57:58.600559 27067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.190:45439 every 8 connection(s)
I20260806 02:57:58.602111 26961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
I20260806 02:57:58.606179 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 26961
I20260806 02:57:58.606376 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1785984707115702-26446-0/minicluster-data/master-0/wal/instance
I20260806 02:57:58.606388 27068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:57:58.606948 26446 external_mini_cluster.cc:1428] Setting key 4ba53d617b3cb16062c321247be6d331
I20260806 02:57:58.621354 27068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c778123ff879448b9222139c48f46175: Bootstrap starting.
Aug 06 02:57:58 dist-test-slave-zj8q krb5kdc[26938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:57:58.625093 27068 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c778123ff879448b9222139c48f46175: Neither blocks nor log segments found. Creating new log.
I20260806 02:57:58.626274 27068 log.cc:824] T 00000000000000000000000000000000 P c778123ff879448b9222139c48f46175: Log is configured to *not* fsync() on all Append() calls
I20260806 02:57:58.629627 27068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c778123ff879448b9222139c48f46175: No bootstrap required, opened a new log
I20260806 02:57:58.630492 27071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:58.608873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52951 (local address 127.25.211.190:45439)
0806 02:57:58.609694 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:58.609705 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:58.609749 (+ 44us) server_negotiation.cc:407] Connection header received
0806 02:57:58.611456 (+ 1707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:58.611481 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:58.611907 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:58.612260 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:58.614091 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:58.615436 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:58.617129 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:58.617856 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:58.617995 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:58.622905 (+ 4910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:58.622915 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:58.622930 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:57:58.622975 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:58.626138 (+ 3163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:58.627015 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:58.627024 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:58.627034 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:58.627264 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:58.627675 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:58.627680 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:58.627683 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:58.628246 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:58.628456 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:58.628950 (+ 494us) server_negotiation.cc:299] Negotiation successful
0806 02:57:58.629282 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":344,"threads_started":1}
I20260806 02:57:58.634384 26446 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:57:58.638343 27068 raft_consensus.cc:348] T 00000000000000000000000000000000 P c778123ff879448b9222139c48f46175 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c778123ff879448b9222139c48f46175" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45439 } }
I20260806 02:57:58.638931 27068 raft_consensus.cc:374] T 00000000000000000000000000000000 P c778123ff879448b9222139c48f46175 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:57:58.639066 27068 raft_consensus.cc:729] T 00000000000000000000000000000000 P c778123ff879448b9222139c48f46175 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c778123ff879448b9222139c48f46175, State: Initialized, Role: FOLLOWER
I20260806 02:57:58.639776 27068 consensus_queue.cc:260] T 00000000000000000000000000000000 P c778123ff879448b9222139c48f46175 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c778123ff879448b9222139c48f46175" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45439 } }
I20260806 02:57:58.640071 27068 raft_consensus.cc:388] T 00000000000000000000000000000000 P c778123ff879448b9222139c48f46175 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:57:58.640214 27068 raft_consensus.cc:482] T 00000000000000000000000000000000 P c778123ff879448b9222139c48f46175 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:57:58.640360 27068 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c778123ff879448b9222139c48f46175 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:57:58.643186 27068 raft_consensus.cc:504] T 00000000000000000000000000000000 P c778123ff879448b9222139c48f46175 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c778123ff879448b9222139c48f46175" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45439 } }
I20260806 02:57:58.643711 27068 leader_election.cc:302] T 00000000000000000000000000000000 P c778123ff879448b9222139c48f46175 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c778123ff879448b9222139c48f46175; no voters:
I20260806 02:57:58.644349 27068 leader_election.cc:288] T 00000000000000000000000000000000 P c778123ff879448b9222139c48f46175 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:57:58.644479 27080 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c778123ff879448b9222139c48f46175 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:57:58.645183 27080 raft_consensus.cc:686] T 00000000000000000000000000000000 P c778123ff879448b9222139c48f46175 [term 1 LEADER]: Becoming Leader. State: Replica: c778123ff879448b9222139c48f46175, State: Running, Role: LEADER
I20260806 02:57:58.645891 27080 consensus_queue.cc:237] T 00000000000000000000000000000000 P c778123ff879448b9222139c48f46175 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c778123ff879448b9222139c48f46175" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45439 } }
I20260806 02:57:58.648041 27080 sys_catalog.cc:455] T 00000000000000000000000000000000 P c778123ff879448b9222139c48f46175 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c778123ff879448b9222139c48f46175. Latest consensus state: current_term: 1 leader_uuid: "c778123ff879448b9222139c48f46175" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c778123ff879448b9222139c48f46175" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45439 } } }
I20260806 02:57:58.649663 27080 sys_catalog.cc:458] T 00000000000000000000000000000000 P c778123ff879448b9222139c48f46175 [sys.catalog]: This master's current role is: LEADER
I20260806 02:57:58.649309 27083 sys_catalog.cc:455] T 00000000000000000000000000000000 P c778123ff879448b9222139c48f46175 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c778123ff879448b9222139c48f46175" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c778123ff879448b9222139c48f46175" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45439 } } }
I20260806 02:57:58.650075 27083 sys_catalog.cc:458] T 00000000000000000000000000000000 P c778123ff879448b9222139c48f46175 [sys.catalog]: This master's current role is: LEADER
I20260806 02:57:58.655265 27071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:57:58.638681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52209 (local address 127.25.211.190:45439)
0806 02:57:58.638838 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 02:57:58.638845 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:57:58.639412 (+ 567us) server_negotiation.cc:407] Connection header received
0806 02:57:58.639610 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:57:58.639616 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:57:58.639699 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:57:58.639814 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:57:58.641378 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:58.642509 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:58.643896 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:57:58.644609 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:57:58.644760 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:57:58.647876 (+ 3116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:57:58.647883 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:57:58.647888 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:57:58.647930 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:57:58.651835 (+ 3905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:58.652858 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:58.652865 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:58.652869 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:58.653041 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:57:58.653476 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:57:58.653480 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:57:58.653483 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:57:58.653944 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:57:58.654290 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0806 02:57:58.654636 (+ 346us) server_negotiation.cc:299] Negotiation successful
0806 02:57:58.654869 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 02:57:58.654907 27068 sys_catalog.cc:564] T 00000000000000000000000000000000 P c778123ff879448b9222139c48f46175 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:57:58.655556 27086 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:57:58.660826 27086 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:57:58.672928 27086 catalog_manager.cc:1347] Generated new cluster ID: c02d066b1bee4439b1c92cbd506af8af
I20260806 02:57:58.673065 27086 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:57:58.686914 27086 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:57:58.688321 27086 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:57:58.696516 27086 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c778123ff879448b9222139c48f46175: Generated new TSK 0
I20260806 02:57:58.697409 27086 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:57:58.707283 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 26961
I20260806 02:57:58.708811 27061 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:57:58.799099 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 26961
2026-08-06T02:57:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2255 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984707115702-26446-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:57:58 dist-test-slave-zj8q krb5kdc[27109](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:57:58 dist-test-slave-zj8q krb5kdc[27109](info): set up 2 sockets
Aug 06 02:57:58 dist-test-slave-zj8q krb5kdc[27109](info): commencing operation
krb5kdc: starting...
W20260806 02:58:00.026446 26446 mini_kdc.cc:121] Time spent starting KDC: real 1.186s user 0.004s sys 0.050s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Aug 06 02:58:00 dist-test-slave-zj8q krb5kdc[27109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:58:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:58:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.211.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab.
Entry for principal kudu/127.25.211.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab.
I20260806 02:58:00.375635 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984707115702-26446-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984707115702-26446-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984707115702-26446-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:45963
--webserver_interface=127.25.211.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:37281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.211.190:45963
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:58:00.730970 27129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:58:00.731305 27129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:58:00.731403 27129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:58:00.739027 27129 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:58:00.739163 27129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:58:00.739221 27129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:58:00.739283 27129 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:58:00.739328 27129 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:58:00.749683 27129 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:37281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984707115702-26446-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984707115702-26446-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.211.190:45963
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:45963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.211.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984707115702-26446-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:58:00.751101 27129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:58:00.752418 27129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:58:00.764200 27135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:58:00.764171 27136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:58:00.764446 27138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:58:00.764962 27129 server_base.cc:1034] running on GCE node
I20260806 02:58:00.765784 27129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:58:00.767321 27129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:58:00.768540 27129 hybrid_clock.cc:648] HybridClock initialized: now 1785985080768488 us; error 44 us; skew 500 ppm
I20260806 02:58:00.768934 27129 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:58:00 dist-test-slave-zj8q krb5kdc[27109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985080, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:58:00.772329 27129 init.cc:377] Logged in from keytab as kudu/127.25.211.190@KRBTEST.COM (short username kudu)
I20260806 02:58:00.774017 27129 webserver.cc:460] Webserver started at http://127.25.211.190:34839/ using document root <none> and password file <none>
I20260806 02:58:00.774847 27129 fs_manager.cc:359] Metadata directory not provided
I20260806 02:58:00.774986 27129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:58:00.775315 27129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:58:00.778831 27129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984707115702-26446-0/minicluster-data/master-0/data/instance:
uuid: "90451965c4fe4ee095a66ac0a104d96b"
format_stamp: "Formatted at 2026-08-06 02:58:00 on dist-test-slave-zj8q"
server_key: "ddff48ebad997bfd134c23aeefab67c0"
server_key_iv: "c9a8f0967537055243cdac5f69aecf3c"
server_key_version: "encryptionkey@0"
I20260806 02:58:00.779685 27129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984707115702-26446-0/minicluster-data/master-0/wal/instance:
uuid: "90451965c4fe4ee095a66ac0a104d96b"
format_stamp: "Formatted at 2026-08-06 02:58:00 on dist-test-slave-zj8q"
server_key: "ddff48ebad997bfd134c23aeefab67c0"
server_key_iv: "c9a8f0967537055243cdac5f69aecf3c"
server_key_version: "encryptionkey@0"
I20260806 02:58:00.785382 27129 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 02:58:00.789409 27149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:58:00.791051 27129 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:58:00.791286 27129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984707115702-26446-0/minicluster-data/master-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984707115702-26446-0/minicluster-data/master-0/wal
uuid: "90451965c4fe4ee095a66ac0a104d96b"
format_stamp: "Formatted at 2026-08-06 02:58:00 on dist-test-slave-zj8q"
server_key: "ddff48ebad997bfd134c23aeefab67c0"
server_key_iv: "c9a8f0967537055243cdac5f69aecf3c"
server_key_version: "encryptionkey@0"
I20260806 02:58:00.791522 27129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984707115702-26446-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984707115702-26446-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984707115702-26446-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:58:00.831658 27129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:58:00.836298 27129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:58:00.836572 27129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:58:00.853812 27129 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.190:45963
I20260806 02:58:00.853832 27235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.190:45963 every 8 connection(s)
I20260806 02:58:00.855238 27129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
I20260806 02:58:00.859174 27236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:58:00.861008 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 27129
I20260806 02:58:00.861220 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1785984707115702-26446-0/minicluster-data/master-0/wal/instance
I20260806 02:58:00.861778 26446 external_mini_cluster.cc:1428] Setting key f7d562c187b351d739660984c5814dea
I20260806 02:58:00.873826 27236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90451965c4fe4ee095a66ac0a104d96b: Bootstrap starting.
I20260806 02:58:00.875417 27241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:00.863767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51751 (local address 127.25.211.190:45963)
0806 02:58:00.864588 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:00.864607 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:00.864652 (+ 45us) server_negotiation.cc:407] Connection header received
0806 02:58:00.866318 (+ 1666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:00.866342 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:00.866763 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:00.867080 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:00.867688 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:00.868968 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:00.870837 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:00.871566 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:00.871723 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:00.873015 (+ 1292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:00.873029 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:00.873048 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:58:00.873618 (+ 570us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:00.873699 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:58:00.873725 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:00.873816 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:00.874074 (+ 258us) server_negotiation.cc:299] Negotiation successful
0806 02:58:00.874405 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":369,"threads_started":1}
I20260806 02:58:00.877242 27236 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90451965c4fe4ee095a66ac0a104d96b: Neither blocks nor log segments found. Creating new log.
I20260806 02:58:00.878479 27236 log.cc:824] T 00000000000000000000000000000000 P 90451965c4fe4ee095a66ac0a104d96b: Log is configured to *not* fsync() on all Append() calls
I20260806 02:58:00.879015 26446 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:58:00.881278 27236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90451965c4fe4ee095a66ac0a104d96b: No bootstrap required, opened a new log
I20260806 02:58:00.890746 27241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:00.883358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48879 (local address 127.25.211.190:45963)
0806 02:58:00.883547 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:00.883553 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:00.884088 (+ 535us) server_negotiation.cc:407] Connection header received
0806 02:58:00.884291 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:00.884295 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:00.884357 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:00.884445 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:00.885035 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:00.885782 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:00.887778 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:00.888337 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:00.888467 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:00.889789 (+ 1322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:00.889796 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:00.889801 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:58:00.889826 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:00.889877 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:58:00.889902 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:00.889995 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:00.890284 (+ 289us) server_negotiation.cc:299] Negotiation successful
0806 02:58:00.890397 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 02:58:00.891636 27236 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90451965c4fe4ee095a66ac0a104d96b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90451965c4fe4ee095a66ac0a104d96b" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45963 } }
I20260806 02:58:00.892060 27236 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90451965c4fe4ee095a66ac0a104d96b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:58:00.892174 27236 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90451965c4fe4ee095a66ac0a104d96b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90451965c4fe4ee095a66ac0a104d96b, State: Initialized, Role: FOLLOWER
I20260806 02:58:00.892845 27236 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90451965c4fe4ee095a66ac0a104d96b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90451965c4fe4ee095a66ac0a104d96b" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45963 } }
I20260806 02:58:00.893086 27236 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90451965c4fe4ee095a66ac0a104d96b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:58:00.893224 27236 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90451965c4fe4ee095a66ac0a104d96b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:58:00.893384 27236 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90451965c4fe4ee095a66ac0a104d96b [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:58:00.895903 27236 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90451965c4fe4ee095a66ac0a104d96b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90451965c4fe4ee095a66ac0a104d96b" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45963 } }
I20260806 02:58:00.896515 27236 leader_election.cc:302] T 00000000000000000000000000000000 P 90451965c4fe4ee095a66ac0a104d96b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90451965c4fe4ee095a66ac0a104d96b; no voters:
I20260806 02:58:00.897230 27236 leader_election.cc:288] T 00000000000000000000000000000000 P 90451965c4fe4ee095a66ac0a104d96b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:58:00.897413 27254 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90451965c4fe4ee095a66ac0a104d96b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:58:00.898147 27254 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90451965c4fe4ee095a66ac0a104d96b [term 1 LEADER]: Becoming Leader. State: Replica: 90451965c4fe4ee095a66ac0a104d96b, State: Running, Role: LEADER
I20260806 02:58:00.898892 27254 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90451965c4fe4ee095a66ac0a104d96b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90451965c4fe4ee095a66ac0a104d96b" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45963 } }
I20260806 02:58:00.899672 27236 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90451965c4fe4ee095a66ac0a104d96b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:58:00.901708 27255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90451965c4fe4ee095a66ac0a104d96b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90451965c4fe4ee095a66ac0a104d96b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90451965c4fe4ee095a66ac0a104d96b" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45963 } } }
I20260806 02:58:00.902051 27255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90451965c4fe4ee095a66ac0a104d96b [sys.catalog]: This master's current role is: LEADER
I20260806 02:58:00.902957 27261 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:58:00.904386 27257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90451965c4fe4ee095a66ac0a104d96b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90451965c4fe4ee095a66ac0a104d96b. Latest consensus state: current_term: 1 leader_uuid: "90451965c4fe4ee095a66ac0a104d96b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90451965c4fe4ee095a66ac0a104d96b" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 45963 } } }
I20260806 02:58:00.904837 27257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90451965c4fe4ee095a66ac0a104d96b [sys.catalog]: This master's current role is: LEADER
I20260806 02:58:00.908568 27261 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:58:00.928314 27261 catalog_manager.cc:1347] Generated new cluster ID: 8eb1760542a2490aa243eb869bba6309
I20260806 02:58:00.928463 27261 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:58:00.949285 27261 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:58:00.950771 27261 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:58:00.958145 27261 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 90451965c4fe4ee095a66ac0a104d96b: Generated new TSK 0
I20260806 02:58:00.959193 27261 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:58:00.971417 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 27129
W20260806 02:58:00.972699 27227 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51751
W20260806 02:58:00.973466 26446 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 27129: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 02:58:00.973671 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 27129
2026-08-06T02:58:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2176 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984707115702-26446-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:58:01 dist-test-slave-zj8q krb5kdc[27281](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:58:01 dist-test-slave-zj8q krb5kdc[27281](info): set up 2 sockets
Aug 06 02:58:01 dist-test-slave-zj8q krb5kdc[27281](info): commencing operation
krb5kdc: starting...
W20260806 02:58:02.209618 26446 mini_kdc.cc:121] Time spent starting KDC: real 1.192s user 0.003s sys 0.051s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Aug 06 02:58:02 dist-test-slave-zj8q krb5kdc[27281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:58:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:58:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.211.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab.
Entry for principal kudu/127.25.211.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab.
I20260806 02:58:02.559619 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984707115702-26446-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984707115702-26446-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984707115702-26446-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:32801
--webserver_interface=127.25.211.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:36241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.211.190:32801
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:58:02.915779 27295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:58:02.916113 27295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:58:02.916247 27295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:58:02.923794 27295 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:58:02.923926 27295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:58:02.923985 27295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:58:02.924077 27295 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:58:02.924131 27295 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:58:02.934641 27295 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:36241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984707115702-26446-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984707115702-26446-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.211.190:32801
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:32801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.211.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984707115702-26446-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:58:02.936048 27295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:58:02.937405 27295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:58:02.949622 27307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:58:02.949704 27303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:58:02.949644 27304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:58:02.950456 27295 server_base.cc:1034] running on GCE node
I20260806 02:58:02.951331 27295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:58:02.952967 27295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:58:02.954234 27295 hybrid_clock.cc:648] HybridClock initialized: now 1785985082954186 us; error 45 us; skew 500 ppm
I20260806 02:58:02.954658 27295 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:58:02 dist-test-slave-zj8q krb5kdc[27281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985082, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:58:02.970319 27295 init.cc:377] Logged in from keytab as kudu/127.25.211.190@KRBTEST.COM (short username kudu)
I20260806 02:58:02.972091 27295 webserver.cc:460] Webserver started at http://127.25.211.190:37211/ using document root <none> and password file <none>
I20260806 02:58:02.972888 27295 fs_manager.cc:359] Metadata directory not provided
I20260806 02:58:02.973019 27295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:58:02.973349 27295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:58:02.976886 27295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984707115702-26446-0/minicluster-data/master-0/data/instance:
uuid: "5f4cfe1636bd4a649a2cecd617b9855a"
format_stamp: "Formatted at 2026-08-06 02:58:02 on dist-test-slave-zj8q"
server_key: "5373a35a1caf76cfe08df4fc969ce83c"
server_key_iv: "d1aafb63905abc7c99a4cf4afef8ce47"
server_key_version: "encryptionkey@0"
I20260806 02:58:02.977706 27295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984707115702-26446-0/minicluster-data/master-0/wal/instance:
uuid: "5f4cfe1636bd4a649a2cecd617b9855a"
format_stamp: "Formatted at 2026-08-06 02:58:02 on dist-test-slave-zj8q"
server_key: "5373a35a1caf76cfe08df4fc969ce83c"
server_key_iv: "d1aafb63905abc7c99a4cf4afef8ce47"
server_key_version: "encryptionkey@0"
I20260806 02:58:02.983767 27295 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260806 02:58:02.987969 27319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:58:02.989601 27295 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260806 02:58:02.989817 27295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984707115702-26446-0/minicluster-data/master-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984707115702-26446-0/minicluster-data/master-0/wal
uuid: "5f4cfe1636bd4a649a2cecd617b9855a"
format_stamp: "Formatted at 2026-08-06 02:58:02 on dist-test-slave-zj8q"
server_key: "5373a35a1caf76cfe08df4fc969ce83c"
server_key_iv: "d1aafb63905abc7c99a4cf4afef8ce47"
server_key_version: "encryptionkey@0"
I20260806 02:58:02.990037 27295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984707115702-26446-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984707115702-26446-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984707115702-26446-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:58:02.996265 27295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:58:03.000748 27295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:58:03.001046 27295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:58:03.019734 27295 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.190:32801
I20260806 02:58:03.019784 27402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.190:32801 every 8 connection(s)
I20260806 02:58:03.021168 27295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
I20260806 02:58:03.024997 27403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:58:03.025139 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 27295
I20260806 02:58:03.025312 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1785984707115702-26446-0/minicluster-data/master-0/wal/instance
I20260806 02:58:03.025756 26446 external_mini_cluster.cc:1428] Setting key 7959897036855ce5caa7ded6bcb6c216
I20260806 02:58:03.034013 27406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:03.027860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48099 (local address 127.25.211.190:32801)
0806 02:58:03.028735 (+ 875us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:03.028747 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:03.028789 (+ 42us) server_negotiation.cc:407] Connection header received
0806 02:58:03.030406 (+ 1617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:03.030431 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:03.030811 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:03.031206 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:03.032295 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:03.032305 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:03.032319 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:58:03.032334 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:03.032411 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:58:03.032435 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:03.032647 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:03.032800 (+ 153us) server_negotiation.cc:299] Negotiation successful
0806 02:58:03.033088 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":375,"threads_started":1}
I20260806 02:58:03.037854 26446 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:58:03.038919 27403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f4cfe1636bd4a649a2cecd617b9855a: Bootstrap starting.
I20260806 02:58:03.042217 27403 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f4cfe1636bd4a649a2cecd617b9855a: Neither blocks nor log segments found. Creating new log.
I20260806 02:58:03.043673 27403 log.cc:824] T 00000000000000000000000000000000 P 5f4cfe1636bd4a649a2cecd617b9855a: Log is configured to *not* fsync() on all Append() calls
I20260806 02:58:03.044615 27406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:03.042097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57407 (local address 127.25.211.190:32801)
0806 02:58:03.042317 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:03.042322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:03.042676 (+ 354us) server_negotiation.cc:407] Connection header received
0806 02:58:03.042871 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:03.042874 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:03.042923 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:03.043002 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:03.043932 (+ 930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:03.043936 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:03.043939 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 02:58:03.043953 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:03.043991 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 02:58:03.044006 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:03.044083 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:03.044260 (+ 177us) server_negotiation.cc:299] Negotiation successful
0806 02:58:03.044323 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260806 02:58:03.046916 27403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f4cfe1636bd4a649a2cecd617b9855a: No bootstrap required, opened a new log
I20260806 02:58:03.055208 27403 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f4cfe1636bd4a649a2cecd617b9855a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f4cfe1636bd4a649a2cecd617b9855a" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 32801 } }
I20260806 02:58:03.055611 27403 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f4cfe1636bd4a649a2cecd617b9855a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 02:58:03.055689 27403 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f4cfe1636bd4a649a2cecd617b9855a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f4cfe1636bd4a649a2cecd617b9855a, State: Initialized, Role: FOLLOWER
I20260806 02:58:03.056239 27403 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f4cfe1636bd4a649a2cecd617b9855a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f4cfe1636bd4a649a2cecd617b9855a" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 32801 } }
I20260806 02:58:03.056461 27403 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f4cfe1636bd4a649a2cecd617b9855a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 02:58:03.056564 27403 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f4cfe1636bd4a649a2cecd617b9855a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 02:58:03.056690 27403 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f4cfe1636bd4a649a2cecd617b9855a [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:58:03.058939 27403 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f4cfe1636bd4a649a2cecd617b9855a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f4cfe1636bd4a649a2cecd617b9855a" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 32801 } }
I20260806 02:58:03.059441 27403 leader_election.cc:302] T 00000000000000000000000000000000 P 5f4cfe1636bd4a649a2cecd617b9855a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f4cfe1636bd4a649a2cecd617b9855a; no voters:
I20260806 02:58:03.060045 27403 leader_election.cc:288] T 00000000000000000000000000000000 P 5f4cfe1636bd4a649a2cecd617b9855a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 02:58:03.060217 27417 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f4cfe1636bd4a649a2cecd617b9855a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:58:03.062310 27403 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f4cfe1636bd4a649a2cecd617b9855a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:58:03.062652 27417 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f4cfe1636bd4a649a2cecd617b9855a [term 1 LEADER]: Becoming Leader. State: Replica: 5f4cfe1636bd4a649a2cecd617b9855a, State: Running, Role: LEADER
I20260806 02:58:03.063215 27417 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f4cfe1636bd4a649a2cecd617b9855a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f4cfe1636bd4a649a2cecd617b9855a" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 32801 } }
I20260806 02:58:03.067348 27418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f4cfe1636bd4a649a2cecd617b9855a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f4cfe1636bd4a649a2cecd617b9855a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f4cfe1636bd4a649a2cecd617b9855a" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 32801 } } }
I20260806 02:58:03.067371 27420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f4cfe1636bd4a649a2cecd617b9855a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f4cfe1636bd4a649a2cecd617b9855a. Latest consensus state: current_term: 1 leader_uuid: "5f4cfe1636bd4a649a2cecd617b9855a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f4cfe1636bd4a649a2cecd617b9855a" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 32801 } } }
I20260806 02:58:03.067881 27418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f4cfe1636bd4a649a2cecd617b9855a [sys.catalog]: This master's current role is: LEADER
I20260806 02:58:03.067934 27420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f4cfe1636bd4a649a2cecd617b9855a [sys.catalog]: This master's current role is: LEADER
I20260806 02:58:03.068701 27434 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:58:03.074795 27434 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:58:03.085201 27434 catalog_manager.cc:1347] Generated new cluster ID: 9a65115b2e66427d9f82f3248639f0f4
I20260806 02:58:03.085345 27434 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:58:03.125627 27434 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:58:03.126122 27434 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:58:03.132808 27434 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f4cfe1636bd4a649a2cecd617b9855a: Generated new TSK 0
I20260806 02:58:03.134016 27434 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:58:03.149820 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 27295
W20260806 02:58:03.151157 27392 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48099
W20260806 02:58:03.151957 26446 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 27295: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 02:58:03.152128 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 27295
2026-08-06T02:58:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2175 ms)
[----------] 18 tests from AuthTokenIssuingTest (39983 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:58:03 dist-test-slave-zj8q krb5kdc[27446](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:58:03 dist-test-slave-zj8q krb5kdc[27446](info): set up 2 sockets
Aug 06 02:58:03 dist-test-slave-zj8q krb5kdc[27446](info): commencing operation
krb5kdc: starting...
W20260806 02:58:04.413398 26446 mini_kdc.cc:121] Time spent starting KDC: real 1.219s user 0.000s sys 0.056s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Aug 06 02:58:04 dist-test-slave-zj8q krb5kdc[27446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985084, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:58:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:58:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.211.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab.
Entry for principal kudu/127.25.211.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab.
I20260806 02:58:04.765552 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:44755
--webserver_interface=127.25.211.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:42617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.211.190:44755,127.25.211.189:36331,127.25.211.188:43657
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:58:05.126461 27462 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 02:58:05.126829 27462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:58:05.126906 27462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:58:05.126989 27462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:58:05.134523 27462 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:58:05.134636 27462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:58:05.134693 27462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:58:05.134744 27462 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:58:05.134807 27462 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:58:05.145118 27462 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:42617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.211.190:44755,127.25.211.189:36331,127.25.211.188:43657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:44755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.211.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:58:05.146530 27462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:58:05.147822 27462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:58:05.159956 27467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:58:05.160068 27469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:58:05.160035 27472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:58:05.161047 27462 server_base.cc:1034] running on GCE node
I20260806 02:58:05.161871 27462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:58:05.163401 27462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:58:05.164649 27462 hybrid_clock.cc:648] HybridClock initialized: now 1785985085164587 us; error 28 us; skew 500 ppm
I20260806 02:58:05.165015 27462 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:58:05 dist-test-slave-zj8q krb5kdc[27446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985085, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:58:05.168321 27462 init.cc:377] Logged in from keytab as kudu/127.25.211.190@KRBTEST.COM (short username kudu)
I20260806 02:58:05.170025 27462 webserver.cc:460] Webserver started at http://127.25.211.190:43519/ using document root <none> and password file <none>
I20260806 02:58:05.171027 27462 fs_manager.cc:359] Metadata directory not provided
I20260806 02:58:05.171170 27462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:58:05.171517 27462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:58:05.174786 27462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/minicluster-data/master-0/data/instance:
uuid: "ccbf13e6d5ad4a189d9c1718b0744272"
format_stamp: "Formatted at 2026-08-06 02:58:05 on dist-test-slave-zj8q"
server_key: "37ed6dc0a7336fc3d6011dac203d89c6"
server_key_iv: "ffc0b7c0588779d5c09b66768680d6e8"
server_key_version: "encryptionkey@0"
I20260806 02:58:05.175630 27462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/minicluster-data/master-0/wal/instance:
uuid: "ccbf13e6d5ad4a189d9c1718b0744272"
format_stamp: "Formatted at 2026-08-06 02:58:05 on dist-test-slave-zj8q"
server_key: "37ed6dc0a7336fc3d6011dac203d89c6"
server_key_iv: "ffc0b7c0588779d5c09b66768680d6e8"
server_key_version: "encryptionkey@0"
I20260806 02:58:05.181330 27462 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 02:58:05.185333 27485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:58:05.186874 27462 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 02:58:05.187074 27462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/minicluster-data/master-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/minicluster-data/master-0/wal
uuid: "ccbf13e6d5ad4a189d9c1718b0744272"
format_stamp: "Formatted at 2026-08-06 02:58:05 on dist-test-slave-zj8q"
server_key: "37ed6dc0a7336fc3d6011dac203d89c6"
server_key_iv: "ffc0b7c0588779d5c09b66768680d6e8"
server_key_version: "encryptionkey@0"
I20260806 02:58:05.187309 27462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:58:05.213860 27462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:58:05.218770 27462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:58:05.219059 27462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:58:05.238286 27462 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.190:44755
I20260806 02:58:05.238306 27562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.190:44755 every 8 connection(s)
I20260806 02:58:05.239777 27462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
I20260806 02:58:05.241768 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 27462
I20260806 02:58:05.241967 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/minicluster-data/master-0/wal/instance
I20260806 02:58:05.242702 26446 external_mini_cluster.cc:1428] Setting key 1dc747ea8d1945e9fc2b37860a17a3ec
I20260806 02:58:05.243865 27566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:58:05.251212 27566 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.211.190" port: 44755 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:58:05 dist-test-slave-zj8q krb5kdc[27446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985085, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.190@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:58:05.276785 27568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:05.254666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:44755 (local address 127.0.0.1:40596)
0806 02:58:05.255633 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:05.255689 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:05.258014 (+ 2325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:05.259168 (+ 1154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:05.259174 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:05.259789 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:58:05.260155 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:05.260174 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:05.262404 (+ 2230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:05.262408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:05.263563 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:05.263612 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:05.264885 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:05.264890 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:05.265023 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:05.265782 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:58:05.265807 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:58:05.268334 (+ 2527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:58:05.272376 (+ 4042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:05.272388 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:05.272400 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:05.272863 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:05.273357 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:05.273360 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:05.273363 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:05.273580 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:05.274686 (+ 1106us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:58:05.274693 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:58:05.274909 (+ 216us) client_negotiation.cc:769] Sending connection context
0806 02:58:05.275148 (+ 239us) client_negotiation.cc:240] Negotiation successful
0806 02:58:05.275622 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"thread_start_us":383,"threads_started":1}
I20260806 02:58:05.276834 27570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:05.254666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40596 (local address 127.25.211.190:44755)
0806 02:58:05.256591 (+ 1925us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:05.256598 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:05.256659 (+ 61us) server_negotiation.cc:407] Connection header received
0806 02:58:05.258505 (+ 1846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:05.258515 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:05.258872 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:05.259116 (+ 244us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:05.260390 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:05.262217 (+ 1827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:05.263763 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:05.264635 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:05.264840 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:05.268596 (+ 3756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:05.268607 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:05.268625 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:05.268662 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:05.272212 (+ 3550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:05.273012 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:05.273022 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:05.273025 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:05.273235 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:05.273690 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:05.273695 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:05.273698 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:05.274516 (+ 818us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:05.274738 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:05.275337 (+ 599us) server_negotiation.cc:299] Negotiation successful
0806 02:58:05.275830 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":136,"server-negotiator.queue_time_us":1717,"spinlock_wait_cycles":29952,"thread_start_us":1452,"threads_started":1}
WARNING: no policy specified for kudu/127.25.211.189@KRBTEST.COM; defaulting to no policy
I20260806 02:58:05.279948 27566 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.211.189" port: 36331 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.189@KRBTEST.COM" created.
I20260806 02:58:05.281318 27568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:05.280730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.189:36331 (local address 127.0.0.1:38406)
0806 02:58:05.280934 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:05.281086 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.189:36331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260806 02:58:05.281628 27486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.211.189:36331: connect: Connection refused (error 111)
I20260806 02:58:05.283252 27568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:05.282862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.189:36331 (local address 127.0.0.1:38418)
0806 02:58:05.283014 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:05.283098 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.189:36331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260806 02:58:05.283453 27566 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.211.189:36331: Network error: Client connection negotiation failed: client connection to 127.25.211.189:36331: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.189.keytab.
Entry for principal kudu/127.25.211.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.189.keytab.
I20260806 02:58:05.319832 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.189:36331
--webserver_interface=127.25.211.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.189.keytab
--principal=kudu/127.25.211.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:42617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.211.190:44755,127.25.211.189:36331,127.25.211.188:43657
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:58:05.332823 27566 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.211.189" port: 36331 } attempt: 1
I20260806 02:58:05.334259 27568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:05.333640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.189:36331 (local address 127.0.0.1:38430)
0806 02:58:05.333875 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:05.334034 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.189:36331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260806 02:58:05.335491 27568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:05.335043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.189:36331 (local address 127.0.0.1:38436)
0806 02:58:05.335224 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:05.335319 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.189:36331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260806 02:58:05.335752 27566 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.211.189:36331: Network error: Client connection negotiation failed: client connection to 127.25.211.189:36331: connect: Connection refused (error 111)
I20260806 02:58:05.404156 27566 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.211.189" port: 36331 } attempt: 2
I20260806 02:58:05.405586 27568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:05.404930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.189:36331 (local address 127.0.0.1:38452)
0806 02:58:05.405253 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:05.405381 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.189:36331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260806 02:58:05.406818 27568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:05.406365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.189:36331 (local address 127.0.0.1:38454)
0806 02:58:05.406530 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:05.406641 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.189:36331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260806 02:58:05.407063 27566 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.211.189:36331: Network error: Client connection negotiation failed: client connection to 127.25.211.189:36331: connect: Connection refused (error 111)
I20260806 02:58:05.498389 27566 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.211.189" port: 36331 } attempt: 3
I20260806 02:58:05.499771 27568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:05.499237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.189:36331 (local address 127.0.0.1:38462)
0806 02:58:05.499471 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:05.499590 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.189:36331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260806 02:58:05.500825 27568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:05.500427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.189:36331 (local address 127.0.0.1:38474)
0806 02:58:05.500594 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:05.500670 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.189:36331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260806 02:58:05.501037 27566 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.211.189:36331: Network error: Client connection negotiation failed: client connection to 127.25.211.189:36331: connect: Connection refused (error 111)
I20260806 02:58:05.644304 27566 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.211.189" port: 36331 } attempt: 4
I20260806 02:58:05.645579 27568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:05.644981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.189:36331 (local address 127.0.0.1:38486)
0806 02:58:05.645332 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:05.645421 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.189:36331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183}
I20260806 02:58:05.646807 27568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:05.646419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.189:36331 (local address 127.0.0.1:38494)
0806 02:58:05.646572 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:05.646670 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.189:36331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260806 02:58:05.647032 27566 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.211.189:36331: Network error: Client connection negotiation failed: client connection to 127.25.211.189:36331: connect: Connection refused (error 111)
W20260806 02:58:05.683992 27580 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 02:58:05.684355 27580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:58:05.684459 27580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:58:05.684620 27580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:58:05.692252 27580 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:58:05.692378 27580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:58:05.692454 27580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:58:05.692529 27580 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:58:05.692611 27580 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:58:05.703078 27580 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:42617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.25.211.190:44755,127.25.211.189:36331,127.25.211.188:43657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.189.keytab
--principal=kudu/127.25.211.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.189:36331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.25.211.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:58:05.704680 27580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:58:05.706239 27580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:58:05.718843 27591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:58:05.718725 27592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:58:05.719085 27594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:58:05.719316 27580 server_base.cc:1034] running on GCE node
I20260806 02:58:05.720381 27580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:58:05.721997 27580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:58:05.723227 27580 hybrid_clock.cc:648] HybridClock initialized: now 1785985085723149 us; error 59 us; skew 500 ppm
I20260806 02:58:05.723577 27580 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:58:05 dist-test-slave-zj8q krb5kdc[27446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985085, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:58:05.726969 27580 init.cc:377] Logged in from keytab as kudu/127.25.211.189@KRBTEST.COM (short username kudu)
I20260806 02:58:05.728534 27580 webserver.cc:460] Webserver started at http://127.25.211.189:43315/ using document root <none> and password file <none>
I20260806 02:58:05.729291 27580 fs_manager.cc:359] Metadata directory not provided
I20260806 02:58:05.729404 27580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:58:05.729740 27580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:58:05.733037 27580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/minicluster-data/master-1/data/instance:
uuid: "576c022c7b8e45c8a515a4777de4d480"
format_stamp: "Formatted at 2026-08-06 02:58:05 on dist-test-slave-zj8q"
server_key: "8fbcb0c6826f6314ac7354e6c89f7967"
server_key_iv: "7c0ce20aa9b237af8d22c9625251fcd6"
server_key_version: "encryptionkey@0"
I20260806 02:58:05.733915 27580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/minicluster-data/master-1/wal/instance:
uuid: "576c022c7b8e45c8a515a4777de4d480"
format_stamp: "Formatted at 2026-08-06 02:58:05 on dist-test-slave-zj8q"
server_key: "8fbcb0c6826f6314ac7354e6c89f7967"
server_key_iv: "7c0ce20aa9b237af8d22c9625251fcd6"
server_key_version: "encryptionkey@0"
I20260806 02:58:05.739739 27580 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260806 02:58:05.743643 27605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:58:05.745144 27580 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 02:58:05.745357 27580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/minicluster-data/master-1/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/minicluster-data/master-1/wal
uuid: "576c022c7b8e45c8a515a4777de4d480"
format_stamp: "Formatted at 2026-08-06 02:58:05 on dist-test-slave-zj8q"
server_key: "8fbcb0c6826f6314ac7354e6c89f7967"
server_key_iv: "7c0ce20aa9b237af8d22c9625251fcd6"
server_key_version: "encryptionkey@0"
I20260806 02:58:05.745622 27580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:58:05.767805 27580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:58:05.772554 27580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:58:05.772853 27580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:58:05.792757 27580 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.189:36331
I20260806 02:58:05.792780 27677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.189:36331 every 8 connection(s)
I20260806 02:58:05.794286 27580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/minicluster-data/master-1/data/info.pb
I20260806 02:58:05.796344 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 27580
I20260806 02:58:05.796543 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/minicluster-data/master-1/wal/instance
I20260806 02:58:05.797137 26446 external_mini_cluster.cc:1428] Setting key a5969aeca845493e86597ecce2b5534d
I20260806 02:58:05.799233 27678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:58:05.808223 27678 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.211.190" port: 44755 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:58:05 dist-test-slave-zj8q krb5kdc[27446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985085, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.189@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.25.211.188@KRBTEST.COM; defaulting to no policy
I20260806 02:58:05.834975 27681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:05.811700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40604 (local address 127.25.211.190:44755)
0806 02:58:05.815848 (+ 4148us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:05.815855 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:05.815882 (+ 27us) server_negotiation.cc:407] Connection header received
0806 02:58:05.817889 (+ 2007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:05.817895 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:05.817970 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:05.818082 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:05.820358 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:05.821138 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:05.823480 (+ 2342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:05.823933 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:05.824044 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:05.828630 (+ 4586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:05.828636 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:05.828640 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:05.828691 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:05.831530 (+ 2839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:05.832390 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:05.832395 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:05.832397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:05.832529 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:05.832992 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:05.832996 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:05.832999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:05.833313 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:05.833455 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:05.834336 (+ 881us) server_negotiation.cc:299] Negotiation successful
0806 02:58:05.834591 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3998,"thread_start_us":3885,"threads_started":1}
I20260806 02:58:05.836047 27682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:05.812266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:44755 (local address 127.0.0.1:40604)
0806 02:58:05.815057 (+ 2791us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:05.815110 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:05.817530 (+ 2420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:05.818456 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:05.818471 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:05.819247 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:58:05.820067 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:05.820098 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:05.821361 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:05.821368 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:05.823285 (+ 1917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:05.823305 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:05.824243 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:05.824252 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:05.824587 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:05.825435 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:58:05.825467 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:58:05.828318 (+ 2851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:58:05.831713 (+ 3395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:05.831723 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:05.831740 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:05.832238 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:05.832646 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:05.832651 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:05.832655 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:05.832863 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:05.833454 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:58:05.833464 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:58:05.833902 (+ 438us) client_negotiation.cc:769] Sending connection context
0806 02:58:05.834198 (+ 296us) client_negotiation.cc:240] Negotiation successful
0806 02:58:05.834766 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2493,"thread_start_us":2259,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.188@KRBTEST.COM" created.
I20260806 02:58:05.837986 27678 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.211.189" port: 36331 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:58:05 dist-test-slave-zj8q krb5kdc[27446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985085, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.189@KRBTEST.COM for kudu/127.25.211.189@KRBTEST.COM
I20260806 02:58:05.852995 27682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:05.838820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.189:36331 (local address 127.0.0.1:38496)
0806 02:58:05.838988 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:05.839004 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:05.839098 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:05.840335 (+ 1237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:05.840340 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:05.840794 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:58:05.840940 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:05.840949 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:05.842450 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:05.842455 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:05.843306 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:05.843318 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:05.844153 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:05.844159 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:05.844343 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:05.844905 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:58:05.844923 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:58:05.847209 (+ 2286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:58:05.850486 (+ 3277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:05.850492 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:05.850496 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:05.850872 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:05.851234 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:05.851239 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:05.851243 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:05.851395 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:05.852021 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:58:05.852025 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:58:05.852170 (+ 145us) client_negotiation.cc:769] Sending connection context
0806 02:58:05.852290 (+ 120us) client_negotiation.cc:240] Negotiation successful
0806 02:58:05.852582 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260806 02:58:05.853078 27684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:05.839263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38496 (local address 127.25.211.189:36331)
0806 02:58:05.839902 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:05.839908 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:05.839936 (+ 28us) server_negotiation.cc:407] Connection header received
0806 02:58:05.839982 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:05.839987 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:05.840177 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:05.840324 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:05.841118 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:05.842319 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:05.843446 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:05.844009 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:05.844125 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:05.847419 (+ 3294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:05.847432 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:05.847448 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:05.847480 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:05.850358 (+ 2878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:05.851001 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:05.851008 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:05.851010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:05.851128 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:05.851502 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:05.851506 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:05.851508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:05.851885 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:05.852096 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:05.852463 (+ 367us) server_negotiation.cc:299] Negotiation successful
0806 02:58:05.852704 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":345,"threads_started":1}
I20260806 02:58:05.855647 27678 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.211.188" port: 43657 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:58:05.857112 27682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:05.856579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:43657 (local address 127.0.0.1:38470)
0806 02:58:05.856781 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:05.856915 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.188:43657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260806 02:58:05.857491 27609 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.211.188:43657: connect: Connection refused (error 111)
I20260806 02:58:05.859370 27682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:05.858961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:43657 (local address 127.0.0.1:38482)
0806 02:58:05.859143 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:05.859234 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.188:43657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260806 02:58:05.859628 27678 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.211.188:43657: Network error: Client connection negotiation failed: client connection to 127.25.211.188:43657: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.188.keytab.
Entry for principal kudu/127.25.211.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.188.keytab.
I20260806 02:58:05.881384 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.188:43657
--webserver_interface=127.25.211.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.188.keytab
--principal=kudu/127.25.211.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:42617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.211.190:44755,127.25.211.189:36331,127.25.211.188:43657
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:58:05.908986 27678 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.211.188" port: 43657 } attempt: 1
I20260806 02:58:05.910271 27682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:05.909771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:43657 (local address 127.0.0.1:38498)
0806 02:58:05.909954 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:05.910068 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.188:43657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260806 02:58:05.911283 27682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:05.910933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:43657 (local address 127.0.0.1:38508)
0806 02:58:05.911095 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:05.911171 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.188:43657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260806 02:58:05.911497 27678 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.211.188:43657: Network error: Client connection negotiation failed: client connection to 127.25.211.188:43657: connect: Connection refused (error 111)
I20260806 02:58:05.946316 27566 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.211.189" port: 36331 } attempt: 5
I20260806 02:58:05.979715 27678 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.211.188" port: 43657 } attempt: 2
I20260806 02:58:05.980894 27682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:05.980417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:43657 (local address 127.0.0.1:38510)
0806 02:58:05.980628 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:05.980720 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.188:43657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260806 02:58:05.982021 27682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:05.981661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:43657 (local address 127.0.0.1:38516)
0806 02:58:05.981814 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:05.981893 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.188:43657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260806 02:58:05.982297 27678 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.211.188:43657: Network error: Client connection negotiation failed: client connection to 127.25.211.188:43657: connect: Connection refused (error 111)
I20260806 02:58:06.073516 27678 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.211.188" port: 43657 } attempt: 3
I20260806 02:58:06.074733 27682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:06.074251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:43657 (local address 127.0.0.1:38522)
0806 02:58:06.074447 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:06.074558 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.188:43657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260806 02:58:06.076112 27682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:06.075586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:43657 (local address 127.0.0.1:38532)
0806 02:58:06.075859 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:06.075950 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.188:43657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171}
W20260806 02:58:06.076370 27678 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.211.188:43657: Network error: Client connection negotiation failed: client connection to 127.25.211.188:43657: connect: Connection refused (error 111)
I20260806 02:58:06.219609 27678 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.211.188" port: 43657 } attempt: 4
I20260806 02:58:06.220875 27682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:06.220389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:43657 (local address 127.0.0.1:38542)
0806 02:58:06.220608 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:06.220706 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.188:43657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260806 02:58:06.221869 27682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:06.221533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:43657 (local address 127.0.0.1:38548)
0806 02:58:06.221675 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:06.221760 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.188:43657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260806 02:58:06.222079 27678 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.211.188:43657: Network error: Client connection negotiation failed: client connection to 127.25.211.188:43657: connect: Connection refused (error 111)
W20260806 02:58:06.253589 27687 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 02:58:06.253952 27687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:58:06.254030 27687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:58:06.254118 27687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:58:06.261762 27687 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:58:06.261878 27687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:58:06.261937 27687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:58:06.261998 27687 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:58:06.262069 27687 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:58:06.272557 27687 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:42617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.25.211.190:44755,127.25.211.189:36331,127.25.211.188:43657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.188.keytab
--principal=kudu/127.25.211.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.188:43657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.25.211.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:58:06.273870 27687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:58:06.275338 27687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:58:06.287443 27700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:58:06.287487 27696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:58:06.287689 27694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:58:06.288496 27687 server_base.cc:1034] running on GCE node
I20260806 02:58:06.289358 27687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:58:06.290920 27687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:58:06.292156 27687 hybrid_clock.cc:648] HybridClock initialized: now 1785985086292104 us; error 33 us; skew 500 ppm
I20260806 02:58:06.292537 27687 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 02:58:06.521337 27678 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.211.188" port: 43657 } attempt: 5
I20260806 02:58:06.522629 27682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:06.522115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:43657 (local address 127.0.0.1:38562)
0806 02:58:06.522340 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:06.522439 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.188:43657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260806 02:58:06.523905 27682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:06.523449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:43657 (local address 127.0.0.1:38576)
0806 02:58:06.523668 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:06.523779 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.188:43657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260806 02:58:06.524169 27678 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.211.188:43657: Network error: Client connection negotiation failed: client connection to 127.25.211.188:43657: connect: Connection refused (error 111)
W20260806 02:58:06.767099 27568 sasl_common.cc:323] Time spent creating new SASL kudu client: real 0.815s user 0.001s sys 0.000s
W20260806 02:58:06.767234 27687 net_util.cc:504] Time spent look up canonical hostname for localhost 'dist-test-slave-zj8q': real 0.474s user 0.001s sys 0.000s
Aug 06 02:58:06 dist-test-slave-zj8q krb5kdc[27446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985085, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.190@KRBTEST.COM for kudu/127.25.211.189@KRBTEST.COM
Aug 06 02:58:06 dist-test-slave-zj8q krb5kdc[27446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985086, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:58:06.770838 27687 init.cc:377] Logged in from keytab as kudu/127.25.211.188@KRBTEST.COM (short username kudu)
I20260806 02:58:06.773116 27687 webserver.cc:460] Webserver started at http://127.25.211.188:46409/ using document root <none> and password file <none>
I20260806 02:58:06.774142 27687 fs_manager.cc:359] Metadata directory not provided
I20260806 02:58:06.774324 27687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:58:06.774730 27687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:58:06.777927 27684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:05.947256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38500 (local address 127.25.211.189:36331)
0806 02:58:05.947486 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:05.947491 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:05.947564 (+ 73us) server_negotiation.cc:407] Connection header received
0806 02:58:05.947764 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:05.947770 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:05.947863 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:05.947959 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:05.948825 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:05.949603 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:05.950949 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:05.951489 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:05.951620 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:06.770866 (+819246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:06.770875 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:06.770881 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:06.770935 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:06.774669 (+ 3734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:06.775758 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:06.775765 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:06.775768 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:06.775945 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:06.776410 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:06.776414 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:06.776416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:06.776733 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:06.776878 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:06.777345 (+ 467us) server_negotiation.cc:299] Negotiation successful
0806 02:58:06.777590 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260806 02:58:06.777951 27568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:05.947213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.189:36331 (local address 127.0.0.1:38500)
0806 02:58:05.947448 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:05.947472 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:05.947630 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:05.948027 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:05.948032 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:05.948462 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:58:05.948650 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:05.948661 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:05.949751 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:05.949755 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:05.950794 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:05.950809 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:05.951667 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:05.951672 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:05.951915 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:06.767287 (+815372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:58:06.767330 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:58:06.770506 (+ 3176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:58:06.774947 (+ 4441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:06.774954 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:06.774958 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:06.775511 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:06.776073 (+ 562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:06.776079 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:06.776083 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:06.776262 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:06.776876 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:58:06.776881 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:58:06.777072 (+ 191us) client_negotiation.cc:769] Sending connection context
0806 02:58:06.777222 (+ 150us) client_negotiation.cc:240] Negotiation successful
0806 02:58:06.777590 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260806 02:58:06.779074 27566 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.211.188" port: 43657 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:58:06.781636 27687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/minicluster-data/master-2/data/instance:
uuid: "aa6e8a2ee2024f0595cbebd637ac5545"
format_stamp: "Formatted at 2026-08-06 02:58:06 on dist-test-slave-zj8q"
server_key: "a90343a994bb77d4e094a305caf34b66"
server_key_iv: "27a1fa8ddce8c512b211fec886248b37"
server_key_version: "encryptionkey@0"
I20260806 02:58:06.782829 27687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/minicluster-data/master-2/wal/instance:
uuid: "aa6e8a2ee2024f0595cbebd637ac5545"
format_stamp: "Formatted at 2026-08-06 02:58:06 on dist-test-slave-zj8q"
server_key: "a90343a994bb77d4e094a305caf34b66"
server_key_iv: "27a1fa8ddce8c512b211fec886248b37"
server_key_version: "encryptionkey@0"
I20260806 02:58:06.783136 27568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:06.782568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:43657 (local address 127.0.0.1:38592)
0806 02:58:06.782860 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:06.782961 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.188:43657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166}
I20260806 02:58:06.788892 27568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:06.788318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:43657 (local address 127.0.0.1:38602)
0806 02:58:06.788532 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:06.788649 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.188:43657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260806 02:58:06.789166 27566 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.211.188:43657: Network error: Client connection negotiation failed: client connection to 127.25.211.188:43657: connect: Connection refused (error 111)
I20260806 02:58:06.791177 27687 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.005s sys 0.003s
I20260806 02:58:06.795737 27711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:58:06.797545 27687 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260806 02:58:06.797792 27687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/minicluster-data/master-2/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/minicluster-data/master-2/wal
uuid: "aa6e8a2ee2024f0595cbebd637ac5545"
format_stamp: "Formatted at 2026-08-06 02:58:06 on dist-test-slave-zj8q"
server_key: "a90343a994bb77d4e094a305caf34b66"
server_key_iv: "27a1fa8ddce8c512b211fec886248b37"
server_key_version: "encryptionkey@0"
I20260806 02:58:06.798048 27687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:58:06.824203 27687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:58:06.829371 27687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:58:06.829721 27687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:58:06.840427 27566 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.211.188" port: 43657 } attempt: 1
I20260806 02:58:06.841722 27568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:06.841163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:43657 (local address 127.0.0.1:38618)
0806 02:58:06.841416 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:06.841527 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.188:43657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260806 02:58:06.842818 27568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:06.842395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:43657 (local address 127.0.0.1:38622)
0806 02:58:06.842553 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:06.842641 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.188:43657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260806 02:58:06.843106 27566 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.211.188:43657: Network error: Client connection negotiation failed: client connection to 127.25.211.188:43657: connect: Connection refused (error 111)
I20260806 02:58:06.849722 27687 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.188:43657
I20260806 02:58:06.849726 27794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.188:43657 every 8 connection(s)
I20260806 02:58:06.851233 27687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/minicluster-data/master-2/data/info.pb
I20260806 02:58:06.855336 27797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:58:06.860565 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 27687
I20260806 02:58:06.860747 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1785984707115702-26446-0/minicluster-data/master-2/wal/instance
I20260806 02:58:06.861148 26446 external_mini_cluster.cc:1428] Setting key 83296983be915dfecabe892fe0d9614c
I20260806 02:58:06.863142 27797 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.211.190" port: 44755 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:58:06 dist-test-slave-zj8q krb5kdc[27446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985084, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.188@KRBTEST.COM
Aug 06 02:58:06 dist-test-slave-zj8q krb5kdc[27446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985086, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.188@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:58:06.885262 27802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:06.863164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38626 (local address 127.25.211.188:43657)
0806 02:58:06.863968 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:06.863981 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:06.864023 (+ 42us) server_negotiation.cc:407] Connection header received
0806 02:58:06.865762 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:06.865795 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:06.866470 (+ 675us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:06.866885 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:06.868427 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:06.869744 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:06.871103 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:06.871993 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:06.872166 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:06.876685 (+ 4519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:06.876693 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:06.876707 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:06.876743 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:06.880234 (+ 3491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:06.881048 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:06.881058 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:06.881068 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:06.881275 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:06.881688 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:06.881693 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:06.881696 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:06.882563 (+ 867us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:06.882916 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:06.883367 (+ 451us) server_negotiation.cc:299] Negotiation successful
0806 02:58:06.883843 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":348,"threads_started":1}
I20260806 02:58:06.885818 27803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:06.866068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40612 (local address 127.25.211.190:44755)
0806 02:58:06.866676 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:06.866682 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:06.870954 (+ 4272us) server_negotiation.cc:407] Connection header received
0806 02:58:06.871157 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:06.871163 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:06.871256 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:06.871360 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:06.872577 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:06.873634 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:06.875066 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:06.875473 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:06.875590 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:06.879362 (+ 3772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:06.879370 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:06.879374 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:06.879422 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:06.882660 (+ 3238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:06.883570 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:06.883576 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:06.883579 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:06.883752 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:06.884148 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:06.884153 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:06.884156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:06.884526 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:06.884673 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:06.885086 (+ 413us) server_negotiation.cc:299] Negotiation successful
0806 02:58:06.885333 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":285,"threads_started":1}
I20260806 02:58:06.885846 27804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:06.866127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:44755 (local address 127.0.0.1:40612)
0806 02:58:06.870784 (+ 4657us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:06.870839 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:06.870998 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:06.871421 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:06.871426 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:06.872123 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:58:06.872409 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:06.872418 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:06.873808 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:06.873812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:06.874911 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:06.874924 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:06.875710 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:06.875716 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:06.875852 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:06.876511 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:58:06.876538 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:58:06.879087 (+ 2549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:58:06.882826 (+ 3739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:06.882836 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:06.882844 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:06.883423 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:06.883875 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:06.883878 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:06.883881 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:06.884031 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:06.884653 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:58:06.884659 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:58:06.884796 (+ 137us) client_negotiation.cc:769] Sending connection context
0806 02:58:06.884938 (+ 142us) client_negotiation.cc:240] Negotiation successful
0806 02:58:06.885336 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4402,"thread_start_us":4202,"threads_started":1}
I20260806 02:58:06.887447 27797 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.211.189" port: 36331 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:58:06 dist-test-slave-zj8q krb5kdc[27446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985086, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.188@KRBTEST.COM for kudu/127.25.211.189@KRBTEST.COM
Aug 06 02:58:06 dist-test-slave-zj8q krb5kdc[27446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985084, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:58:06.902815 27804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:06.888206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.189:36331 (local address 127.0.0.1:38504)
0806 02:58:06.888470 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:06.888486 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:06.888599 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:06.888939 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:06.888944 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:06.889441 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:58:06.889595 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:06.889604 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:06.891027 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:06.891032 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:06.892342 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:06.892354 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:06.893323 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:06.893328 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:06.893480 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:06.894071 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:58:06.894094 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:58:06.896725 (+ 2631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:58:06.900022 (+ 3297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:06.900026 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:06.900029 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:06.900545 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:06.901005 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:06.901008 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:06.901011 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:06.901203 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:06.901759 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:58:06.901764 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:58:06.901948 (+ 184us) client_negotiation.cc:769] Sending connection context
0806 02:58:06.902110 (+ 162us) client_negotiation.cc:240] Negotiation successful
0806 02:58:06.902456 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177}
I20260806 02:58:06.902850 27684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:06.888209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38504 (local address 127.25.211.189:36331)
0806 02:58:06.888390 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:06.888395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:06.888571 (+ 176us) server_negotiation.cc:407] Connection header received
0806 02:58:06.888720 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:06.888725 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:06.888808 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:06.888906 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:06.889769 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:06.890867 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:06.892487 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:06.893178 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:06.893316 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:06.896968 (+ 3652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:06.896976 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:06.896982 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:06.897041 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:06.899878 (+ 2837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:06.900696 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:06.900701 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:06.900703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:06.900874 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:06.901319 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:06.901323 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:06.901325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:06.901620 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:06.901747 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:06.902331 (+ 584us) server_negotiation.cc:299] Negotiation successful
0806 02:58:06.902507 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 02:58:06.904536 27797 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.211.188" port: 43657 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:58:06.906821 27803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:06.888703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40616 (local address 127.25.211.190:44755)
0806 02:58:06.888856 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:06.888862 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:06.888932 (+ 70us) server_negotiation.cc:407] Connection header received
0806 02:58:06.889105 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:06.889111 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:06.889200 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:06.889312 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:06.890890 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:06.891985 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:06.894017 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:06.894671 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:06.894801 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:06.899406 (+ 4605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:06.899412 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:06.899416 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:06.899451 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:06.902600 (+ 3149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:06.904282 (+ 1682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:06.904287 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:06.904289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:06.904470 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:06.904920 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:06.904924 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:06.904927 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:06.905263 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:06.905500 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:06.906370 (+ 870us) server_negotiation.cc:299] Negotiation successful
0806 02:58:06.906574 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 02:58:06.911357 27566 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.211.188" port: 43657 } attempt: 2
Aug 06 02:58:06 dist-test-slave-zj8q krb5kdc[27446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985086, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.188@KRBTEST.COM for kudu/127.25.211.188@KRBTEST.COM
Aug 06 02:58:06 dist-test-slave-zj8q krb5kdc[27446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985084, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.189@KRBTEST.COM
I20260806 02:58:06.921519 27804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:06.905471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:43657 (local address 127.0.0.1:38634)
0806 02:58:06.905665 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:06.905680 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:06.905763 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:06.906122 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:06.906127 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:06.906602 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:58:06.906742 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:06.906751 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:06.907895 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:06.907900 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:06.908866 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:06.908881 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:06.909867 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:06.909874 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:06.910105 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:06.910808 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:58:06.910826 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:58:06.913287 (+ 2461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:58:06.918065 (+ 4778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:06.918071 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:06.918076 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:06.918730 (+ 654us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:06.919340 (+ 610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:06.919346 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:06.919350 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:06.919662 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:06.920363 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:58:06.920368 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:58:06.920521 (+ 153us) client_negotiation.cc:769] Sending connection context
0806 02:58:06.920743 (+ 222us) client_negotiation.cc:240] Negotiation successful
0806 02:58:06.921106 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260806 02:58:06.922026 27802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:06.905746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38634 (local address 127.25.211.188:43657)
0806 02:58:06.905896 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:06.905901 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:06.905913 (+ 12us) server_negotiation.cc:407] Connection header received
0806 02:58:06.905946 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:06.905949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:06.906000 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:06.906074 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:06.906875 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:06.907778 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:06.909012 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:06.909687 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:06.909811 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:06.913471 (+ 3660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:06.913476 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:06.913479 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:06.913507 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:06.917844 (+ 4337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:06.918842 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:06.918847 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:06.918851 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:06.918997 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:06.919850 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:06.919856 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:06.919859 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:06.920224 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:06.920367 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:06.921114 (+ 747us) server_negotiation.cc:299] Negotiation successful
0806 02:58:06.921428 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
Aug 06 02:58:06 dist-test-slave-zj8q krb5kdc[27446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985085, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.190@KRBTEST.COM for kudu/127.25.211.188@KRBTEST.COM
I20260806 02:58:06.928004 27684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:06.908577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38512 (local address 127.25.211.189:36331)
0806 02:58:06.908795 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:06.908802 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:06.908820 (+ 18us) server_negotiation.cc:407] Connection header received
0806 02:58:06.908876 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:06.908881 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:06.908952 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:06.909043 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:06.910671 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:06.911786 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:06.913657 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:06.914301 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:06.914417 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:06.920028 (+ 5611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:06.920037 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:06.920042 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:06.920097 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:06.924660 (+ 4563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:06.925701 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:06.925708 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:06.925711 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:06.925905 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:06.926413 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:06.926418 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:06.926421 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:06.926823 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:06.927127 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:06.927434 (+ 307us) server_negotiation.cc:299] Negotiation successful
0806 02:58:06.927667 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 02:58:06.928519 27797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa6e8a2ee2024f0595cbebd637ac5545: Bootstrap starting.
I20260806 02:58:06.932878 26446 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:58:06.933739 27797 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa6e8a2ee2024f0595cbebd637ac5545: Neither blocks nor log segments found. Creating new log.
I20260806 02:58:06.935181 27797 log.cc:824] T 00000000000000000000000000000000 P aa6e8a2ee2024f0595cbebd637ac5545: Log is configured to *not* fsync() on all Append() calls
I20260806 02:58:06.936847 27568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:06.912180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:43657 (local address 127.0.0.1:38644)
0806 02:58:06.912443 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:06.912461 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:06.912566 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:06.919335 (+ 6769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:06.919343 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:06.920060 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:58:06.920385 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:06.920403 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:06.922239 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:06.922248 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:06.923500 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:06.923528 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:06.924539 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:06.924547 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:06.924771 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:06.925339 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:58:06.925358 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:58:06.928581 (+ 3223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:58:06.933457 (+ 4876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:06.933463 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:06.933468 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:06.934135 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:06.934606 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:06.934611 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:06.934615 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:06.934789 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:06.935507 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:58:06.935512 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:58:06.935916 (+ 404us) client_negotiation.cc:769] Sending connection context
0806 02:58:06.936101 (+ 185us) client_negotiation.cc:240] Negotiation successful
0806 02:58:06.936439 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150}
I20260806 02:58:06.936908 27808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:06.912279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38644 (local address 127.25.211.188:43657)
0806 02:58:06.918819 (+ 6540us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:06.918825 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:06.918852 (+ 27us) server_negotiation.cc:407] Connection header received
0806 02:58:06.918915 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:06.918919 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:06.919004 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:06.919138 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:06.920604 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:06.921982 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:06.923703 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:06.924345 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:06.924483 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:06.928943 (+ 4460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:06.928952 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:06.928957 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:06.928999 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:06.933285 (+ 4286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:06.934287 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:06.934293 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:06.934296 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:06.934490 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:06.934937 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:06.934942 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:06.934945 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:06.935377 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:06.935524 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:06.936227 (+ 703us) server_negotiation.cc:299] Negotiation successful
0806 02:58:06.936470 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6381,"thread_start_us":6123,"threads_started":1}
I20260806 02:58:06.939074 27797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa6e8a2ee2024f0595cbebd637ac5545: No bootstrap required, opened a new log
I20260806 02:58:06.936305 27521 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "ccbf13e6d5ad4a189d9c1718b0744272"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40616
I20260806 02:58:06.943670 27566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccbf13e6d5ad4a189d9c1718b0744272: Bootstrap starting.
I20260806 02:58:06.948130 27566 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccbf13e6d5ad4a189d9c1718b0744272: Neither blocks nor log segments found. Creating new log.
I20260806 02:58:06.949596 27566 log.cc:824] T 00000000000000000000000000000000 P ccbf13e6d5ad4a189d9c1718b0744272: Log is configured to *not* fsync() on all Append() calls
I20260806 02:58:06.949743 27797 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa6e8a2ee2024f0595cbebd637ac5545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccbf13e6d5ad4a189d9c1718b0744272" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 44755 } } peers { permanent_uuid: "576c022c7b8e45c8a515a4777de4d480" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 36331 } } peers { permanent_uuid: "aa6e8a2ee2024f0595cbebd637ac5545" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 43657 } }
I20260806 02:58:06.950268 27797 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa6e8a2ee2024f0595cbebd637ac5545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa6e8a2ee2024f0595cbebd637ac5545, State: Initialized, Role: FOLLOWER
I20260806 02:58:06.950784 27797 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa6e8a2ee2024f0595cbebd637ac5545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccbf13e6d5ad4a189d9c1718b0744272" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 44755 } } peers { permanent_uuid: "576c022c7b8e45c8a515a4777de4d480" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 36331 } } peers { permanent_uuid: "aa6e8a2ee2024f0595cbebd637ac5545" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 43657 } }
I20260806 02:58:06.951829 27812 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa6e8a2ee2024f0595cbebd637ac5545 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccbf13e6d5ad4a189d9c1718b0744272" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 44755 } } peers { permanent_uuid: "576c022c7b8e45c8a515a4777de4d480" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 36331 } } peers { permanent_uuid: "aa6e8a2ee2024f0595cbebd637ac5545" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 43657 } } }
I20260806 02:58:06.952258 27812 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa6e8a2ee2024f0595cbebd637ac5545 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:58:06.953235 27797 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa6e8a2ee2024f0595cbebd637ac5545 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:58:06.953758 27566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccbf13e6d5ad4a189d9c1718b0744272: No bootstrap required, opened a new log
I20260806 02:58:06.961848 27566 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccbf13e6d5ad4a189d9c1718b0744272 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccbf13e6d5ad4a189d9c1718b0744272" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 44755 } } peers { permanent_uuid: "576c022c7b8e45c8a515a4777de4d480" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 36331 } } peers { permanent_uuid: "aa6e8a2ee2024f0595cbebd637ac5545" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 43657 } }
I20260806 02:58:06.962282 27566 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccbf13e6d5ad4a189d9c1718b0744272 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccbf13e6d5ad4a189d9c1718b0744272, State: Initialized, Role: FOLLOWER
I20260806 02:58:06.962837 27566 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccbf13e6d5ad4a189d9c1718b0744272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccbf13e6d5ad4a189d9c1718b0744272" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 44755 } } peers { permanent_uuid: "576c022c7b8e45c8a515a4777de4d480" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 36331 } } peers { permanent_uuid: "aa6e8a2ee2024f0595cbebd637ac5545" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 43657 } }
W20260806 02:58:06.963550 27831 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aa6e8a2ee2024f0595cbebd637ac5545: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:58:06.963682 27831 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:58:06.964102 27833 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccbf13e6d5ad4a189d9c1718b0744272 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccbf13e6d5ad4a189d9c1718b0744272" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 44755 } } peers { permanent_uuid: "576c022c7b8e45c8a515a4777de4d480" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 36331 } } peers { permanent_uuid: "aa6e8a2ee2024f0595cbebd637ac5545" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 43657 } } }
I20260806 02:58:06.964534 27833 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccbf13e6d5ad4a189d9c1718b0744272 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:58:06.965197 27566 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccbf13e6d5ad4a189d9c1718b0744272 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:58:06.965633 27521 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccbf13e6d5ad4a189d9c1718b0744272 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 02:58:06.965854 27521 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccbf13e6d5ad4a189d9c1718b0744272 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:58:06.969396 27521 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccbf13e6d5ad4a189d9c1718b0744272 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccbf13e6d5ad4a189d9c1718b0744272" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 44755 } } peers { permanent_uuid: "576c022c7b8e45c8a515a4777de4d480" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 36331 } } peers { permanent_uuid: "aa6e8a2ee2024f0595cbebd637ac5545" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 43657 } }
I20260806 02:58:06.971704 27521 leader_election.cc:288] T 00000000000000000000000000000000 P ccbf13e6d5ad4a189d9c1718b0744272 [CANDIDATE]: Term 1 election: Requested vote from peers 576c022c7b8e45c8a515a4777de4d480 (127.25.211.189:36331), aa6e8a2ee2024f0595cbebd637ac5545 (127.25.211.188:43657)
W20260806 02:58:06.972819 27493 tablet.cc:2369] T 00000000000000000000000000000000 P ccbf13e6d5ad4a189d9c1718b0744272: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:58:06.972640 27752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ccbf13e6d5ad4a189d9c1718b0744272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "aa6e8a2ee2024f0595cbebd637ac5545"
I20260806 02:58:06.973207 27752 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa6e8a2ee2024f0595cbebd637ac5545 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:58:06.976830 27752 raft_consensus.cc:2454] T 00000000000000000000000000000000 P aa6e8a2ee2024f0595cbebd637ac5545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccbf13e6d5ad4a189d9c1718b0744272 in term 1.
I20260806 02:58:06.977705 27489 leader_election.cc:302] T 00000000000000000000000000000000 P ccbf13e6d5ad4a189d9c1718b0744272 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa6e8a2ee2024f0595cbebd637ac5545, ccbf13e6d5ad4a189d9c1718b0744272; no voters:
I20260806 02:58:06.978281 27833 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccbf13e6d5ad4a189d9c1718b0744272 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:58:06.978955 27833 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccbf13e6d5ad4a189d9c1718b0744272 [term 1 LEADER]: Becoming Leader. State: Replica: ccbf13e6d5ad4a189d9c1718b0744272, State: Running, Role: LEADER
I20260806 02:58:06.961522 27687 master_runner.cc:186] Error getting master registration for 127.25.211.190:44755: OK, instance_id { permanent_uuid: "ccbf13e6d5ad4a189d9c1718b0744272" instance_seqno: 1785985085232585 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 02:58:06.979693 27833 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccbf13e6d5ad4a189d9c1718b0744272 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccbf13e6d5ad4a189d9c1718b0744272" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 44755 } } peers { permanent_uuid: "576c022c7b8e45c8a515a4777de4d480" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 36331 } } peers { permanent_uuid: "aa6e8a2ee2024f0595cbebd637ac5545" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 43657 } }
I20260806 02:58:06.979969 27687 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 02:58:06.996098 27842 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccbf13e6d5ad4a189d9c1718b0744272 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccbf13e6d5ad4a189d9c1718b0744272. Latest consensus state: current_term: 1 leader_uuid: "ccbf13e6d5ad4a189d9c1718b0744272" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccbf13e6d5ad4a189d9c1718b0744272" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 44755 } } peers { permanent_uuid: "576c022c7b8e45c8a515a4777de4d480" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 36331 } } peers { permanent_uuid: "aa6e8a2ee2024f0595cbebd637ac5545" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 43657 } } }
I20260806 02:58:06.996619 27842 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccbf13e6d5ad4a189d9c1718b0744272 [sys.catalog]: This master's current role is: LEADER
I20260806 02:58:06.997929 27857 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:58:07.019333 27857 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:58:07.025102 27684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:06.985656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38516 (local address 127.25.211.189:36331)
0806 02:58:06.985865 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:06.985871 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:06.994222 (+ 8351us) server_negotiation.cc:407] Connection header received
0806 02:58:06.994715 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:06.994721 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:06.994809 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:06.994877 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:06.998665 (+ 3788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:06.999788 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:07.002967 (+ 3179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:07.003704 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:07.003809 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:07.017022 (+ 13213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:07.017030 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:07.017035 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:07.017084 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:07.020814 (+ 3730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:07.021822 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:07.021828 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:07.021832 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:07.022036 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:07.022698 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:07.022704 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:07.022708 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:07.023119 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:07.023395 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:07.024451 (+ 1056us) server_negotiation.cc:299] Negotiation successful
0806 02:58:07.024737 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 02:58:07.029711 27803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:06.985427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40618 (local address 127.25.211.190:44755)
0806 02:58:06.985644 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:06.985651 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:06.993024 (+ 7373us) server_negotiation.cc:407] Connection header received
0806 02:58:06.993964 (+ 940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:06.993971 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:06.994064 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:06.994166 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:07.004870 (+ 10704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:07.006172 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:07.010828 (+ 4656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:07.011552 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:07.011683 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:07.020597 (+ 8914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:07.020606 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:07.020612 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:07.020668 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:07.025469 (+ 4801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:07.026494 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:07.026501 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:07.026505 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:07.026685 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:07.027499 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:07.027505 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:07.027509 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:07.027958 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:07.028288 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:07.028918 (+ 630us) server_negotiation.cc:299] Negotiation successful
0806 02:58:07.029253 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 02:58:07.046412 27808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:06.986526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38652 (local address 127.25.211.188:43657)
0806 02:58:06.986735 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:06.986743 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:07.010439 (+ 23696us) server_negotiation.cc:407] Connection header received
0806 02:58:07.013098 (+ 2659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:07.013105 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:07.013223 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:07.013298 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:07.015788 (+ 2490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:07.016866 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:07.032637 (+ 15771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:07.033468 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:07.033590 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:07.037880 (+ 4290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:07.037889 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:07.037894 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:07.037950 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:07.042241 (+ 4291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:07.043285 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:07.043296 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:07.043300 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:07.043485 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:07.044165 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:07.044172 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:07.044176 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:07.044646 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:07.044915 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:07.045616 (+ 701us) server_negotiation.cc:299] Negotiation successful
0806 02:58:07.045906 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 02:58:07.059652 27752 raft_consensus.cc:1261] T 00000000000000000000000000000000 P aa6e8a2ee2024f0595cbebd637ac5545 [term 1 FOLLOWER]: Refusing update from remote peer ccbf13e6d5ad4a189d9c1718b0744272: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:58:07.061144 27833 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ccbf13e6d5ad4a189d9c1718b0744272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa6e8a2ee2024f0595cbebd637ac5545" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 43657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:58:07.071367 27678 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.211.188" port: 43657 } attempt: 6
W20260806 02:58:07.081991 27719 tablet.cc:2369] T 00000000000000000000000000000000 P aa6e8a2ee2024f0595cbebd637ac5545: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:58:07.070446 27462 master_runner.cc:186] Error getting master registration for 127.25.211.189:36331: OK, instance_id { permanent_uuid: "576c022c7b8e45c8a515a4777de4d480" instance_seqno: 1785985085786449 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 02:58:07.085670 27462 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Aug 06 02:58:07 dist-test-slave-zj8q krb5kdc[27446](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985085, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.189@KRBTEST.COM for kudu/127.25.211.188@KRBTEST.COM
I20260806 02:58:07.099609 27808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:07.072425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38662 (local address 127.25.211.188:43657)
0806 02:58:07.072620 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:07.072626 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:07.077799 (+ 5173us) server_negotiation.cc:407] Connection header received
0806 02:58:07.078826 (+ 1027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:07.078832 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:07.078933 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:07.079063 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:07.081212 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:07.082390 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:07.085474 (+ 3084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:07.086278 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:07.086405 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:07.090835 (+ 4430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:07.090842 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:07.090847 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:07.090886 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:07.094657 (+ 3771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:07.095623 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:07.095629 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:07.095665 (+ 36us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:07.095869 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:07.096520 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:07.096526 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:07.096529 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:07.097190 (+ 661us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:07.097557 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:07.098557 (+ 1000us) server_negotiation.cc:299] Negotiation successful
0806 02:58:07.098986 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 02:58:07.099866 27812 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa6e8a2ee2024f0595cbebd637ac5545 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccbf13e6d5ad4a189d9c1718b0744272. Latest consensus state: current_term: 1 leader_uuid: "ccbf13e6d5ad4a189d9c1718b0744272" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccbf13e6d5ad4a189d9c1718b0744272" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 44755 } } peers { permanent_uuid: "576c022c7b8e45c8a515a4777de4d480" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 36331 } } peers { permanent_uuid: "aa6e8a2ee2024f0595cbebd637ac5545" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 43657 } } }
I20260806 02:58:07.100159 27812 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa6e8a2ee2024f0595cbebd637ac5545 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:58:07.100101 27865 mvcc.cc:205] Tried to move back new op lower bound from 7315394916581756928 to 7315394916274745344. Current Snapshot: MvccSnapshot[applied={T|T < 7315394916581756928}]
I20260806 02:58:07.101147 27875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:07.071974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:43657 (local address 127.0.0.1:38662)
0806 02:58:07.077634 (+ 5660us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:07.077664 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:07.077852 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:07.079453 (+ 1601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:07.079459 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:07.080514 (+ 1055us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:58:07.080920 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:07.080962 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:07.082771 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:07.082776 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:07.085175 (+ 2399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:07.085194 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:07.086480 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:07.086488 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:07.086813 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:07.087582 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:58:07.087627 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:58:07.090595 (+ 2968us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:58:07.094853 (+ 4258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:07.094860 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:07.094865 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:07.095471 (+ 606us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:07.096043 (+ 572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:07.096049 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:07.096053 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:07.096316 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:07.097710 (+ 1394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:58:07.097715 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:58:07.098006 (+ 291us) client_negotiation.cc:769] Sending connection context
0806 02:58:07.100021 (+ 2015us) client_negotiation.cc:240] Negotiation successful
0806 02:58:07.100536 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5497,"thread_start_us":5365,"threads_started":1}
I20260806 02:58:07.109512 27678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 576c022c7b8e45c8a515a4777de4d480: Bootstrap starting.
I20260806 02:58:07.110672 27874 mvcc.cc:205] Tried to move back new op lower bound from 7315394916581756928 to 7315394916274745344. Current Snapshot: MvccSnapshot[applied={T|T < 7315394916581756928}]
I20260806 02:58:07.115300 27812 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa6e8a2ee2024f0595cbebd637ac5545 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ccbf13e6d5ad4a189d9c1718b0744272" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccbf13e6d5ad4a189d9c1718b0744272" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 44755 } } peers { permanent_uuid: "576c022c7b8e45c8a515a4777de4d480" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 36331 } } peers { permanent_uuid: "aa6e8a2ee2024f0595cbebd637ac5545" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 43657 } } }
I20260806 02:58:07.115579 27812 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa6e8a2ee2024f0595cbebd637ac5545 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:58:07.117569 27833 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccbf13e6d5ad4a189d9c1718b0744272 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ccbf13e6d5ad4a189d9c1718b0744272" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccbf13e6d5ad4a189d9c1718b0744272" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 44755 } } peers { permanent_uuid: "576c022c7b8e45c8a515a4777de4d480" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 36331 } } peers { permanent_uuid: "aa6e8a2ee2024f0595cbebd637ac5545" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 43657 } } }
I20260806 02:58:07.117887 27833 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccbf13e6d5ad4a189d9c1718b0744272 [sys.catalog]: This master's current role is: LEADER
I20260806 02:58:07.119526 27678 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 576c022c7b8e45c8a515a4777de4d480: Neither blocks nor log segments found. Creating new log.
I20260806 02:58:07.120970 27678 log.cc:824] T 00000000000000000000000000000000 P 576c022c7b8e45c8a515a4777de4d480: Log is configured to *not* fsync() on all Append() calls
I20260806 02:58:07.131042 27857 catalog_manager.cc:1347] Generated new cluster ID: 69fe193aecd949bbb02577c174485dab
I20260806 02:58:07.131181 27857 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:58:07.133140 27678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 576c022c7b8e45c8a515a4777de4d480: No bootstrap required, opened a new log
I20260806 02:58:07.144930 27678 raft_consensus.cc:348] T 00000000000000000000000000000000 P 576c022c7b8e45c8a515a4777de4d480 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccbf13e6d5ad4a189d9c1718b0744272" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 44755 } } peers { permanent_uuid: "576c022c7b8e45c8a515a4777de4d480" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 36331 } } peers { permanent_uuid: "aa6e8a2ee2024f0595cbebd637ac5545" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 43657 } }
I20260806 02:58:07.145394 27678 raft_consensus.cc:729] T 00000000000000000000000000000000 P 576c022c7b8e45c8a515a4777de4d480 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 576c022c7b8e45c8a515a4777de4d480, State: Initialized, Role: FOLLOWER
I20260806 02:58:07.145954 27678 consensus_queue.cc:260] T 00000000000000000000000000000000 P 576c022c7b8e45c8a515a4777de4d480 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccbf13e6d5ad4a189d9c1718b0744272" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 44755 } } peers { permanent_uuid: "576c022c7b8e45c8a515a4777de4d480" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 36331 } } peers { permanent_uuid: "aa6e8a2ee2024f0595cbebd637ac5545" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 43657 } }
I20260806 02:58:07.147274 27882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 576c022c7b8e45c8a515a4777de4d480 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccbf13e6d5ad4a189d9c1718b0744272" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 44755 } } peers { permanent_uuid: "576c022c7b8e45c8a515a4777de4d480" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 36331 } } peers { permanent_uuid: "aa6e8a2ee2024f0595cbebd637ac5545" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 43657 } } }
I20260806 02:58:07.148069 27882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 576c022c7b8e45c8a515a4777de4d480 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:58:07.148583 27678 sys_catalog.cc:564] T 00000000000000000000000000000000 P 576c022c7b8e45c8a515a4777de4d480 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:58:07.151877 27639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ccbf13e6d5ad4a189d9c1718b0744272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "576c022c7b8e45c8a515a4777de4d480"
I20260806 02:58:07.152447 27639 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 576c022c7b8e45c8a515a4777de4d480 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:58:07.157822 27857 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:58:07.159857 27857 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:58:07.161844 27639 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 576c022c7b8e45c8a515a4777de4d480 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccbf13e6d5ad4a189d9c1718b0744272 in term 1.
I20260806 02:58:07.162766 27636 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 576c022c7b8e45c8a515a4777de4d480 [term 1 FOLLOWER]: Refusing update from remote peer ccbf13e6d5ad4a189d9c1718b0744272: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:58:07.164963 27833 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ccbf13e6d5ad4a189d9c1718b0744272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "576c022c7b8e45c8a515a4777de4d480" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 36331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:58:07.193980 27857 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ccbf13e6d5ad4a189d9c1718b0744272: Generated new TSK 0
I20260806 02:58:07.195237 27857 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 02:58:07.248540 27882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 576c022c7b8e45c8a515a4777de4d480 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccbf13e6d5ad4a189d9c1718b0744272. Latest consensus state: current_term: 1 leader_uuid: "ccbf13e6d5ad4a189d9c1718b0744272" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccbf13e6d5ad4a189d9c1718b0744272" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 44755 } } peers { permanent_uuid: "576c022c7b8e45c8a515a4777de4d480" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 36331 } } peers { permanent_uuid: "aa6e8a2ee2024f0595cbebd637ac5545" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 43657 } } }
I20260806 02:58:07.248966 27882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 576c022c7b8e45c8a515a4777de4d480 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:58:07.249593 27897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 576c022c7b8e45c8a515a4777de4d480 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ccbf13e6d5ad4a189d9c1718b0744272" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccbf13e6d5ad4a189d9c1718b0744272" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 44755 } } peers { permanent_uuid: "576c022c7b8e45c8a515a4777de4d480" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 36331 } } peers { permanent_uuid: "aa6e8a2ee2024f0595cbebd637ac5545" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 43657 } } }
I20260806 02:58:07.250291 27897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 576c022c7b8e45c8a515a4777de4d480 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:58:07.252105 27842 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccbf13e6d5ad4a189d9c1718b0744272 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ccbf13e6d5ad4a189d9c1718b0744272" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ccbf13e6d5ad4a189d9c1718b0744272" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 44755 } } peers { permanent_uuid: "576c022c7b8e45c8a515a4777de4d480" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 36331 } } peers { permanent_uuid: "aa6e8a2ee2024f0595cbebd637ac5545" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 43657 } } }
I20260806 02:58:07.252449 27842 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccbf13e6d5ad4a189d9c1718b0744272 [sys.catalog]: This master's current role is: LEADER
I20260806 02:58:07.259953 27906 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260806 02:58:07.327271 26446 mini_kdc.cc:348] Time spent kdestroy: real 0.112s user 0.000s sys 0.036s
W20260806 02:58:07.345745 27803 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:58:07.336333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40626 (local address 127.25.211.190:44755)
0806 02:58:07.336548 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:07.336554 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:07.342885 (+ 6331us) server_negotiation.cc:407] Connection header received
0806 02:58:07.343218 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:07.343225 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:07.343319 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:07.343399 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:07.345284 (+ 1885us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40626: BlockingRecv error: recv got EOF from 127.0.0.1:40626 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260806 02:58:07.350956 27684 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:58:07.336352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38524 (local address 127.25.211.189:36331)
0806 02:58:07.336542 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:07.336547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:07.348555 (+ 12008us) server_negotiation.cc:407] Connection header received
0806 02:58:07.349095 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:07.349102 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:07.349207 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:07.349295 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:07.350709 (+ 1414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38524: BlockingRecv error: recv got EOF from 127.0.0.1:38524 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260806 02:58:07.353302 27808 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:58:07.342682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38672 (local address 127.25.211.188:43657)
0806 02:58:07.342886 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:07.342892 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:07.350911 (+ 8019us) server_negotiation.cc:407] Connection header received
0806 02:58:07.351345 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:07.351351 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:07.351438 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:07.351512 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:07.353017 (+ 1505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38672: BlockingRecv error: recv got EOF from 127.0.0.1:38672 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260806 02:58:07.361413 27684 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:58:07.358691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38536 (local address 127.25.211.189:36331)
0806 02:58:07.358884 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:07.358891 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:07.359680 (+ 789us) server_negotiation.cc:407] Connection header received
0806 02:58:07.359915 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:07.359921 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:07.360048 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:07.360160 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:07.361107 (+ 947us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38536: BlockingRecv error: recv got EOF from 127.0.0.1:38536 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260806 02:58:07.374900 27684 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:58:07.372684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38546 (local address 127.25.211.189:36331)
0806 02:58:07.372884 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:07.372891 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:07.373305 (+ 414us) server_negotiation.cc:407] Connection header received
0806 02:58:07.373543 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:07.373549 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:07.373647 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:07.373754 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:07.374676 (+ 922us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38546: BlockingRecv error: recv got EOF from 127.0.0.1:38546 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260806 02:58:07.386871 27684 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:58:07.384392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38562 (local address 127.25.211.189:36331)
0806 02:58:07.384583 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:07.384590 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:07.385161 (+ 571us) server_negotiation.cc:407] Connection header received
0806 02:58:07.385375 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:07.385382 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:07.385492 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:07.385626 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:07.386615 (+ 989us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38562: BlockingRecv error: recv got EOF from 127.0.0.1:38562 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260806 02:58:07.397102 27684 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:58:07.394733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38576 (local address 127.25.211.189:36331)
0806 02:58:07.394962 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:07.394968 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:07.395515 (+ 547us) server_negotiation.cc:407] Connection header received
0806 02:58:07.395727 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:07.395733 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:07.395831 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:07.395942 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:07.396847 (+ 905us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38576: BlockingRecv error: recv got EOF from 127.0.0.1:38576 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
W20260806 02:58:07.411623 27684 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:58:07.409137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38586 (local address 127.25.211.189:36331)
0806 02:58:07.409338 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:07.409345 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:07.409980 (+ 635us) server_negotiation.cc:407] Connection header received
0806 02:58:07.410239 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:07.410245 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:07.410345 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:07.410453 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:07.411382 (+ 929us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38586: BlockingRecv error: recv got EOF from 127.0.0.1:38586 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260806 02:58:07.434492 27684 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:58:07.432055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38600 (local address 127.25.211.189:36331)
0806 02:58:07.432260 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:07.432266 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:07.432796 (+ 530us) server_negotiation.cc:407] Connection header received
0806 02:58:07.433008 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:07.433014 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:07.433120 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:07.433240 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:07.434248 (+ 1008us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38600: BlockingRecv error: recv got EOF from 127.0.0.1:38600 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260806 02:58:07.475430 27684 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:58:07.472870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38610 (local address 127.25.211.189:36331)
0806 02:58:07.473079 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:07.473086 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:07.473695 (+ 609us) server_negotiation.cc:407] Connection header received
0806 02:58:07.473898 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:07.473905 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:07.474018 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:07.474184 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:07.475149 (+ 965us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38610: BlockingRecv error: recv got EOF from 127.0.0.1:38610 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260806 02:58:07.545440 27684 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:58:07.543431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38624 (local address 127.25.211.189:36331)
0806 02:58:07.543630 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:07.543635 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:07.543957 (+ 322us) server_negotiation.cc:407] Connection header received
0806 02:58:07.544176 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:07.544181 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:07.544259 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:07.544348 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:07.545201 (+ 853us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38624: BlockingRecv error: recv got EOF from 127.0.0.1:38624 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260806 02:58:07.679240 27684 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:58:07.677216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38626 (local address 127.25.211.189:36331)
0806 02:58:07.677467 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:07.677473 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:07.677794 (+ 321us) server_negotiation.cc:407] Connection header received
0806 02:58:07.677991 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:07.677995 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:07.678071 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:07.678192 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:07.678966 (+ 774us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38626: BlockingRecv error: recv got EOF from 127.0.0.1:38626 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
W20260806 02:58:07.940742 27684 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:58:07.939013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38632 (local address 127.25.211.189:36331)
0806 02:58:07.939266 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:07.939270 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:07.939378 (+ 108us) server_negotiation.cc:407] Connection header received
0806 02:58:07.939560 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:07.939564 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:07.939638 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:07.939724 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:07.940523 (+ 799us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38632: BlockingRecv error: recv got EOF from 127.0.0.1:38632 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
I20260806 02:58:07.965822 27831 catalog_manager.cc:1259] Loaded cluster ID: 69fe193aecd949bbb02577c174485dab
I20260806 02:58:07.965953 27831 catalog_manager.cc:1552] T 00000000000000000000000000000000 P aa6e8a2ee2024f0595cbebd637ac5545: loading cluster ID for follower catalog manager: success
I20260806 02:58:07.969131 27831 catalog_manager.cc:1574] T 00000000000000000000000000000000 P aa6e8a2ee2024f0595cbebd637ac5545: acquiring CA information for follower catalog manager: success
I20260806 02:58:07.971060 27831 catalog_manager.cc:1602] T 00000000000000000000000000000000 P aa6e8a2ee2024f0595cbebd637ac5545: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 02:58:08.264951 27906 catalog_manager.cc:1259] Loaded cluster ID: 69fe193aecd949bbb02577c174485dab
I20260806 02:58:08.265074 27906 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 576c022c7b8e45c8a515a4777de4d480: loading cluster ID for follower catalog manager: success
I20260806 02:58:08.267882 27906 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 576c022c7b8e45c8a515a4777de4d480: acquiring CA information for follower catalog manager: success
I20260806 02:58:08.269820 27906 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 576c022c7b8e45c8a515a4777de4d480: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 02:58:08.464193 28021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:08.456998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38640 (local address 127.25.211.189:36331)
0806 02:58:08.457827 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:08.457832 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:08.457851 (+ 19us) server_negotiation.cc:407] Connection header received
0806 02:58:08.457918 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:08.457922 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:08.458069 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:08.458215 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:58:08.458772 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:08.459666 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:08.461678 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:08.462141 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:08.462262 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:08.462861 (+ 599us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:58:08.463454 (+ 593us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:58:08.463605 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:08.463784 (+ 179us) server_negotiation.cc:299] Negotiation successful
0806 02:58:08.463872 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":696,"thread_start_us":542,"threads_started":1}
I20260806 02:58:08.483961 28035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:08.476520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38686 (local address 127.25.211.188:43657)
0806 02:58:08.477051 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:08.477057 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:08.477404 (+ 347us) server_negotiation.cc:407] Connection header received
0806 02:58:08.477778 (+ 374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:08.477783 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:08.477945 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:08.478117 (+ 172us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:58:08.478706 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:08.479462 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:08.481448 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:08.481872 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:08.481965 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:08.482680 (+ 715us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:58:08.483276 (+ 596us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:58:08.483398 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:08.483538 (+ 140us) server_negotiation.cc:299] Negotiation successful
0806 02:58:08.483645 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":267,"threads_started":1}
I20260806 02:58:08.507812 28035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:08.493006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38698 (local address 127.25.211.188:43657)
0806 02:58:08.493229 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:08.493235 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:08.500243 (+ 7008us) server_negotiation.cc:407] Connection header received
0806 02:58:08.500442 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:08.500448 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:08.500563 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:08.500650 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:58:08.501825 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:08.502870 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:08.505356 (+ 2486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:08.506098 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:08.506261 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:08.506964 (+ 703us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:58:08.507126 (+ 162us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:58:08.507186 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:08.507423 (+ 237us) server_negotiation.cc:299] Negotiation successful
0806 02:58:08.507512 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260806 02:58:08.508000 28021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:08.492425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38652 (local address 127.25.211.189:36331)
0806 02:58:08.492707 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:08.492715 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:08.500095 (+ 7380us) server_negotiation.cc:407] Connection header received
0806 02:58:08.500623 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:08.500630 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:08.500744 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:08.500809 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:58:08.501812 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:08.502966 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:08.505497 (+ 2531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:08.506270 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:08.506374 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:08.507133 (+ 759us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:58:08.507271 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:58:08.507340 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:08.507578 (+ 238us) server_negotiation.cc:299] Negotiation successful
0806 02:58:08.507671 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260806 02:58:08.508658 28040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:08.492367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40642 (local address 127.25.211.190:44755)
0806 02:58:08.500765 (+ 8398us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:08.500772 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:08.500790 (+ 18us) server_negotiation.cc:407] Connection header received
0806 02:58:08.500912 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:08.500918 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:08.501125 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:08.501324 (+ 199us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:58:08.502283 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:08.503302 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:08.505652 (+ 2350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:08.506337 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:08.506452 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:08.507221 (+ 769us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:58:08.507694 (+ 473us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:58:08.507809 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:08.508273 (+ 464us) server_negotiation.cc:299] Negotiation successful
0806 02:58:08.508357 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8269,"thread_start_us":8395,"threads_started":1}
I20260806 02:58:08.519044 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 27462
I20260806 02:58:08.520203 27554 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:58:08.655388 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 27462
W20260806 02:58:08.689030 27712 connection.cc:535] server connection from 127.0.0.1:38644 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 02:58:08.689749 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 27580
I20260806 02:58:08.690879 27668 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:58:08.800038 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 27580
I20260806 02:58:08.827025 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 27687
I20260806 02:58:08.828148 27787 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:58:08.938236 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 27687
2026-08-06T02:58:08Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5791 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 02:58:09 dist-test-slave-zj8q krb5kdc[28052](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 02:58:09 dist-test-slave-zj8q krb5kdc[28052](info): set up 2 sockets
Aug 06 02:58:09 dist-test-slave-zj8q krb5kdc[28052](info): commencing operation
krb5kdc: starting...
W20260806 02:58:10.189926 26446 mini_kdc.cc:121] Time spent starting KDC: real 1.206s user 0.001s sys 0.055s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/krb5kdc/test-user.keytab.
Aug 06 02:58:10 dist-test-slave-zj8q krb5kdc[28052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T02:58:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T02:58:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.25.211.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab.
Entry for principal kudu/127.25.211.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab.
I20260806 02:58:10.555682 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:46465
--webserver_interface=127.25.211.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:42775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.211.190:46465,127.25.211.189:33735,127.25.211.188:46387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 02:58:10.920434 28071 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 02:58:10.920794 28071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:58:10.920895 28071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:58:10.921020 28071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:58:10.929296 28071 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:58:10.929404 28071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:58:10.929462 28071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:58:10.929514 28071 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:58:10.929579 28071 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:58:10.939980 28071 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:42775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.211.190:46465,127.25.211.189:33735,127.25.211.188:46387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.190.keytab
--principal=kudu/127.25.211.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.190:46465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.211.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:58:10.941335 28071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:58:10.942857 28071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:58:10.955490 28078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:58:10.955528 28082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:58:10.955449 28084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:58:10.956372 28071 server_base.cc:1034] running on GCE node
I20260806 02:58:10.957254 28071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:58:10.959010 28071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:58:10.960382 28071 hybrid_clock.cc:648] HybridClock initialized: now 1785985090960305 us; error 46 us; skew 500 ppm
I20260806 02:58:10.960773 28071 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:58:10 dist-test-slave-zj8q krb5kdc[28052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985090, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:58:10.964464 28071 init.cc:377] Logged in from keytab as kudu/127.25.211.190@KRBTEST.COM (short username kudu)
I20260806 02:58:10.966450 28071 webserver.cc:460] Webserver started at http://127.25.211.190:44063/ using document root <none> and password file <none>
I20260806 02:58:10.967247 28071 fs_manager.cc:359] Metadata directory not provided
I20260806 02:58:10.967361 28071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:58:10.967700 28071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:58:10.971009 28071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/minicluster-data/master-0/data/instance:
uuid: "efeebd1d291141d8a6251e0b2fbd2314"
format_stamp: "Formatted at 2026-08-06 02:58:10 on dist-test-slave-zj8q"
server_key: "522ac5fc09d582854795b4998f3a3682"
server_key_iv: "a7d2554eefad2c2b757a34c077a22829"
server_key_version: "encryptionkey@0"
I20260806 02:58:10.971930 28071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/minicluster-data/master-0/wal/instance:
uuid: "efeebd1d291141d8a6251e0b2fbd2314"
format_stamp: "Formatted at 2026-08-06 02:58:10 on dist-test-slave-zj8q"
server_key: "522ac5fc09d582854795b4998f3a3682"
server_key_iv: "a7d2554eefad2c2b757a34c077a22829"
server_key_version: "encryptionkey@0"
I20260806 02:58:10.978125 28071 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.003s
I20260806 02:58:10.982137 28093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:58:10.983860 28071 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260806 02:58:10.984066 28071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/minicluster-data/master-0/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/minicluster-data/master-0/wal
uuid: "efeebd1d291141d8a6251e0b2fbd2314"
format_stamp: "Formatted at 2026-08-06 02:58:10 on dist-test-slave-zj8q"
server_key: "522ac5fc09d582854795b4998f3a3682"
server_key_iv: "a7d2554eefad2c2b757a34c077a22829"
server_key_version: "encryptionkey@0"
I20260806 02:58:10.984324 28071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:58:11.012084 28071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:58:11.017369 28071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:58:11.017629 28071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:58:11.036721 28071 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.190:46465
I20260806 02:58:11.036770 28183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.190:46465 every 8 connection(s)
I20260806 02:58:11.038242 28071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/minicluster-data/master-0/data/info.pb
I20260806 02:58:11.042227 28185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:58:11.043699 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 28071
I20260806 02:58:11.043896 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/minicluster-data/master-0/wal/instance
I20260806 02:58:11.044325 26446 external_mini_cluster.cc:1428] Setting key 7800efd623ffa8af6dbf9eb3a5101ca8
I20260806 02:58:11.049186 28185 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.211.190" port: 46465 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:58:11 dist-test-slave-zj8q krb5kdc[28052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985090, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.190@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:58:11.075367 28189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:11.052605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40106 (local address 127.25.211.190:46465)
0806 02:58:11.054411 (+ 1806us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:11.054419 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:11.054474 (+ 55us) server_negotiation.cc:407] Connection header received
0806 02:58:11.056226 (+ 1752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:11.056234 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:11.056576 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:11.056804 (+ 228us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:11.058133 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:11.059852 (+ 1719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:11.061976 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:11.062889 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:11.063097 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:11.066756 (+ 3659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:11.066768 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:11.066787 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:11.066831 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:11.070536 (+ 3705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:11.071341 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:11.071350 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:11.071353 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:11.071602 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:11.072207 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:11.072212 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:11.072214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:11.072981 (+ 767us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:11.073205 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:11.073820 (+ 615us) server_negotiation.cc:299] Negotiation successful
0806 02:58:11.074324 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":160,"server-negotiator.queue_time_us":1588,"spinlock_wait_cycles":41728,"thread_start_us":1354,"threads_started":1}
I20260806 02:58:11.075420 28188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:11.052610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:46465 (local address 127.0.0.1:40106)
0806 02:58:11.053622 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:11.053692 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:11.055756 (+ 2064us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:11.056860 (+ 1104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:11.056866 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:11.057513 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:58:11.057892 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:11.057911 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:11.060045 (+ 2134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:11.060051 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:11.061780 (+ 1729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:11.061796 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:11.063156 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:11.063162 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:11.063362 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:11.064085 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:58:11.064117 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:58:11.066474 (+ 2357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:58:11.070706 (+ 4232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:11.070718 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:11.070730 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:11.071184 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:11.071789 (+ 605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:11.071793 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:11.071796 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:11.071975 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:11.073188 (+ 1213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:58:11.073195 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:58:11.073413 (+ 218us) client_negotiation.cc:769] Sending connection context
0806 02:58:11.073651 (+ 238us) client_negotiation.cc:240] Negotiation successful
0806 02:58:11.074101 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"mutex_wait_us":39,"thread_start_us":395,"threads_started":1}
I20260806 02:58:11.079355 28185 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.211.189" port: 33735 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:58:11.080737 28188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:11.080269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.189:33735 (local address 127.0.0.1:47678)
0806 02:58:11.080472 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:11.080597 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.189:33735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
WARNING: no policy specified for kudu/127.25.211.189@KRBTEST.COM; defaulting to no policy
W20260806 02:58:11.081198 28101 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.211.189:33735: connect: Connection refused (error 111)
I20260806 02:58:11.082931 28188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:11.082536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.189:33735 (local address 127.0.0.1:47684)
0806 02:58:11.082729 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:11.082801 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.189:33735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260806 02:58:11.083146 28185 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.211.189:33735: Network error: Client connection negotiation failed: client connection to 127.25.211.189:33735: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.189.keytab.
Entry for principal kudu/127.25.211.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.189.keytab.
I20260806 02:58:11.123446 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.189:33735
--webserver_interface=127.25.211.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.189.keytab
--principal=kudu/127.25.211.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:42775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.211.190:46465,127.25.211.189:33735,127.25.211.188:46387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:58:11.132577 28185 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.211.189" port: 33735 } attempt: 1
I20260806 02:58:11.133879 28188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:11.133371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.189:33735 (local address 127.0.0.1:47686)
0806 02:58:11.133579 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:11.133735 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.189:33735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260806 02:58:11.134898 28188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:11.134534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.189:33735 (local address 127.0.0.1:47702)
0806 02:58:11.134692 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:11.134761 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.189:33735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260806 02:58:11.135098 28185 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.211.189:33735: Network error: Client connection negotiation failed: client connection to 127.25.211.189:33735: connect: Connection refused (error 111)
I20260806 02:58:11.203343 28185 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.211.189" port: 33735 } attempt: 2
I20260806 02:58:11.204651 28188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:11.204142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.189:33735 (local address 127.0.0.1:47716)
0806 02:58:11.204402 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:11.204505 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.189:33735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260806 02:58:11.205768 28188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:11.205380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.189:33735 (local address 127.0.0.1:47722)
0806 02:58:11.205507 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:11.205605 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.189:33735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260806 02:58:11.205979 28185 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.211.189:33735: Network error: Client connection negotiation failed: client connection to 127.25.211.189:33735: connect: Connection refused (error 111)
I20260806 02:58:11.297261 28185 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.211.189" port: 33735 } attempt: 3
I20260806 02:58:11.298460 28188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:11.297976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.189:33735 (local address 127.0.0.1:47730)
0806 02:58:11.298225 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:11.298319 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.189:33735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260806 02:58:11.299659 28188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:11.299290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.189:33735 (local address 127.0.0.1:47738)
0806 02:58:11.299428 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:11.299505 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.189:33735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260806 02:58:11.299894 28185 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.211.189:33735: Network error: Client connection negotiation failed: client connection to 127.25.211.189:33735: connect: Connection refused (error 111)
I20260806 02:58:11.443184 28185 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.211.189" port: 33735 } attempt: 4
I20260806 02:58:11.444454 28188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:11.443876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.189:33735 (local address 127.0.0.1:47740)
0806 02:58:11.444220 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:11.444313 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.189:33735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177}
I20260806 02:58:11.445626 28188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:11.445267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.189:33735 (local address 127.0.0.1:47746)
0806 02:58:11.445398 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:11.445472 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.189:33735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260806 02:58:11.445868 28185 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.211.189:33735: Network error: Client connection negotiation failed: client connection to 127.25.211.189:33735: connect: Connection refused (error 111)
W20260806 02:58:11.488188 28193 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 02:58:11.488564 28193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:58:11.488641 28193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:58:11.488722 28193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:58:11.496361 28193 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:58:11.496459 28193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:58:11.496515 28193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:58:11.496573 28193 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:58:11.496635 28193 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:58:11.506563 28193 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:42775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.25.211.190:46465,127.25.211.189:33735,127.25.211.188:46387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.189.keytab
--principal=kudu/127.25.211.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.189:33735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.25.211.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:58:11.507834 28193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:58:11.509171 28193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:58:11.521003 28200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:58:11.521088 28199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:58:11.521324 28202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:58:11.522367 28193 server_base.cc:1034] running on GCE node
I20260806 02:58:11.523164 28193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:58:11.524612 28193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:58:11.525853 28193 hybrid_clock.cc:648] HybridClock initialized: now 1785985091525783 us; error 45 us; skew 500 ppm
I20260806 02:58:11.526248 28193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:58:11 dist-test-slave-zj8q krb5kdc[28052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985091, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:58:11.529469 28193 init.cc:377] Logged in from keytab as kudu/127.25.211.189@KRBTEST.COM (short username kudu)
I20260806 02:58:11.531162 28193 webserver.cc:460] Webserver started at http://127.25.211.189:35931/ using document root <none> and password file <none>
I20260806 02:58:11.531893 28193 fs_manager.cc:359] Metadata directory not provided
I20260806 02:58:11.531991 28193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:58:11.532284 28193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:58:11.535611 28193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/minicluster-data/master-1/data/instance:
uuid: "98f47ebe8c2441609fbdf31c49633403"
format_stamp: "Formatted at 2026-08-06 02:58:11 on dist-test-slave-zj8q"
server_key: "769df6f762b2bb3ea0292860d491233a"
server_key_iv: "40bf55ec61416137f1a4eb23e285cfb8"
server_key_version: "encryptionkey@0"
I20260806 02:58:11.536473 28193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/minicluster-data/master-1/wal/instance:
uuid: "98f47ebe8c2441609fbdf31c49633403"
format_stamp: "Formatted at 2026-08-06 02:58:11 on dist-test-slave-zj8q"
server_key: "769df6f762b2bb3ea0292860d491233a"
server_key_iv: "40bf55ec61416137f1a4eb23e285cfb8"
server_key_version: "encryptionkey@0"
I20260806 02:58:11.542296 28193 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260806 02:58:11.546192 28211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:58:11.547677 28193 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:58:11.547847 28193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/minicluster-data/master-1/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/minicluster-data/master-1/wal
uuid: "98f47ebe8c2441609fbdf31c49633403"
format_stamp: "Formatted at 2026-08-06 02:58:11 on dist-test-slave-zj8q"
server_key: "769df6f762b2bb3ea0292860d491233a"
server_key_iv: "40bf55ec61416137f1a4eb23e285cfb8"
server_key_version: "encryptionkey@0"
I20260806 02:58:11.548099 28193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:58:11.564294 28193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:58:11.569051 28193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:58:11.569306 28193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:58:11.587710 28193 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.189:33735
I20260806 02:58:11.587718 28294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.189:33735 every 8 connection(s)
I20260806 02:58:11.589313 28193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/minicluster-data/master-1/data/info.pb
I20260806 02:58:11.593415 28295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:58:11.599257 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 28193
I20260806 02:58:11.599458 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/minicluster-data/master-1/wal/instance
I20260806 02:58:11.599911 26446 external_mini_cluster.cc:1428] Setting key 5cb7dcdd489891148a03024afebb0910
I20260806 02:58:11.600704 28295 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.211.190" port: 46465 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:58:11 dist-test-slave-zj8q krb5kdc[28052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985091, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.189@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:58:11.622316 28297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:11.603577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40108 (local address 127.25.211.190:46465)
0806 02:58:11.604119 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:11.604124 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:11.605178 (+ 1054us) server_negotiation.cc:407] Connection header received
0806 02:58:11.607266 (+ 2088us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:11.607272 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:11.607357 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:11.607453 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:11.609195 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:11.609968 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:11.611638 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:11.612077 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:11.612193 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:11.616258 (+ 4065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:11.616263 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:11.616266 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:11.616304 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:11.619000 (+ 2696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:11.619845 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:11.619850 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:11.619852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:11.619982 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:11.620418 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:11.620421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:11.620423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:11.620731 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:11.620871 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:11.621647 (+ 776us) server_negotiation.cc:299] Negotiation successful
0806 02:58:11.621895 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":294,"threads_started":1}
I20260806 02:58:11.623327 28298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:11.604236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:46465 (local address 127.0.0.1:40108)
0806 02:58:11.605009 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:11.605054 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:11.606932 (+ 1878us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:11.607754 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:11.607764 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:11.608343 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:58:11.608958 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:11.608979 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:11.610147 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:11.610185 (+ 38us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:11.611471 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:11.611485 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:11.612338 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:11.612348 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:11.612577 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:11.613276 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:58:11.613307 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:58:11.615967 (+ 2660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:58:11.619177 (+ 3210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:11.619188 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:11.619207 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:11.619691 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:11.620096 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:11.620101 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:11.620105 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:11.620295 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:11.620871 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:58:11.620880 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:58:11.621259 (+ 379us) client_negotiation.cc:769] Sending connection context
0806 02:58:11.621515 (+ 256us) client_negotiation.cc:240] Negotiation successful
0806 02:58:11.622084 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":319,"threads_started":1}
I20260806 02:58:11.625092 28295 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.211.189" port: 33735 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.25.211.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.25.211.188@KRBTEST.COM" created.
Aug 06 02:58:11 dist-test-slave-zj8q krb5kdc[28052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985091, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.189@KRBTEST.COM for kudu/127.25.211.189@KRBTEST.COM
I20260806 02:58:11.643397 28298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:11.625984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.189:33735 (local address 127.0.0.1:47752)
0806 02:58:11.626187 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:11.626206 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:11.626335 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:11.627264 (+ 929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:11.627267 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:11.627658 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:58:11.627828 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:11.627836 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:11.629832 (+ 1996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:11.629836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:11.630765 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:11.630777 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:11.631983 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:11.631988 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:11.632189 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:11.632784 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:58:11.632802 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:58:11.636691 (+ 3889us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:58:11.640392 (+ 3701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:11.640397 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:11.640401 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:11.640905 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:11.641334 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:11.641338 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:11.641341 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:11.641506 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:11.642390 (+ 884us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:58:11.642394 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:58:11.642539 (+ 145us) client_negotiation.cc:769] Sending connection context
0806 02:58:11.642681 (+ 142us) client_negotiation.cc:240] Negotiation successful
0806 02:58:11.642984 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":56}
I20260806 02:58:11.643447 28303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:11.626034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47752 (local address 127.25.211.189:33735)
0806 02:58:11.626679 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:11.626687 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:11.626720 (+ 33us) server_negotiation.cc:407] Connection header received
0806 02:58:11.626777 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:11.626784 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:11.627068 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:11.627241 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:11.628006 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:11.629665 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:11.630916 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:11.631812 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:11.631972 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:11.636953 (+ 4981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:11.636968 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:11.636994 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:11.637053 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:11.640200 (+ 3147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:11.641067 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:11.641075 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:11.641077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:11.641223 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:11.641666 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:11.641671 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:11.641673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:11.642259 (+ 586us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:11.642441 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:11.642839 (+ 398us) server_negotiation.cc:299] Negotiation successful
0806 02:58:11.643102 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":466,"thread_start_us":308,"threads_started":1}
I20260806 02:58:11.645900 28295 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.211.188" port: 46387 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:58:11.647346 28298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:11.646819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:46387 (local address 127.0.0.1:41256)
0806 02:58:11.647021 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:11.647150 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.188:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260806 02:58:11.647688 28212 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.211.188:46387: connect: Connection refused (error 111)
I20260806 02:58:11.652956 28298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:11.652422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:46387 (local address 127.0.0.1:41262)
0806 02:58:11.652660 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:11.652775 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.188:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260806 02:58:11.653290 28295 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.211.188:46387: Network error: Client connection negotiation failed: client connection to 127.25.211.188:46387: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.25.211.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.188.keytab.
Entry for principal kudu/127.25.211.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.188.keytab.
I20260806 02:58:11.679034 26446 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
/tmp/dist-test-taskFsNc4K/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.188:46387
--webserver_interface=127.25.211.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.188.keytab
--principal=kudu/127.25.211.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.25.211.148:42775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.211.190:46465,127.25.211.189:33735,127.25.211.188:46387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 02:58:11.702666 28295 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.211.188" port: 46387 } attempt: 1
I20260806 02:58:11.704016 28298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:11.703463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:46387 (local address 127.0.0.1:41268)
0806 02:58:11.703699 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:11.703828 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.188:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260806 02:58:11.705269 28298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:11.704818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:46387 (local address 127.0.0.1:41278)
0806 02:58:11.705014 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:11.705128 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.188:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260806 02:58:11.705575 28295 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.211.188:46387: Network error: Client connection negotiation failed: client connection to 127.25.211.188:46387: connect: Connection refused (error 111)
I20260806 02:58:11.745169 28185 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.211.189" port: 33735 } attempt: 5
Aug 06 02:58:11 dist-test-slave-zj8q krb5kdc[28052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985090, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.190@KRBTEST.COM for kudu/127.25.211.189@KRBTEST.COM
I20260806 02:58:11.760262 28188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:11.746001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.189:33735 (local address 127.0.0.1:47764)
0806 02:58:11.746272 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:11.746294 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:11.746419 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:11.746789 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:11.746793 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:11.747204 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:58:11.747362 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:11.747371 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:11.748764 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:11.748768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:11.749678 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:11.749691 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:11.750709 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:11.750716 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:11.750943 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:11.751588 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:58:11.751611 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:58:11.753976 (+ 2365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:58:11.757775 (+ 3799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:11.757779 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:11.757783 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:11.758247 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:11.758650 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:11.758653 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:11.758656 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:11.758800 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:11.759447 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:58:11.759451 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:58:11.759612 (+ 161us) client_negotiation.cc:769] Sending connection context
0806 02:58:11.759732 (+ 120us) client_negotiation.cc:240] Negotiation successful
0806 02:58:11.759995 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
I20260806 02:58:11.760419 28303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:11.746238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47764 (local address 127.25.211.189:33735)
0806 02:58:11.746444 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:11.746451 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:11.746475 (+ 24us) server_negotiation.cc:407] Connection header received
0806 02:58:11.746561 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:11.746566 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:11.746650 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:11.746756 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:11.747524 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:11.748608 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:11.749833 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:11.750535 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:11.750668 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:11.754252 (+ 3584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:11.754260 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:11.754264 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:11.754308 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:11.757580 (+ 3272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:11.758371 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:11.758377 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:11.758381 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:11.758542 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:11.758898 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:11.758903 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:11.758906 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:11.759309 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:11.759491 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:11.759870 (+ 379us) server_negotiation.cc:299] Negotiation successful
0806 02:58:11.760123 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 02:58:11.761413 28185 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.211.188" port: 46387 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:58:11.762542 28188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:11.762136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:46387 (local address 127.0.0.1:41294)
0806 02:58:11.762326 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:11.762407 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.188:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260806 02:58:11.763471 28188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:11.763130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:46387 (local address 127.0.0.1:41298)
0806 02:58:11.763280 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:11.763363 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.188:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260806 02:58:11.763677 28185 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.211.188:46387: Network error: Client connection negotiation failed: client connection to 127.25.211.188:46387: connect: Connection refused (error 111)
I20260806 02:58:11.773837 28295 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.211.188" port: 46387 } attempt: 2
I20260806 02:58:11.775130 28298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:11.774546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:46387 (local address 127.0.0.1:41304)
0806 02:58:11.774790 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:11.774912 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.188:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260806 02:58:11.776372 28298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:11.775909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:46387 (local address 127.0.0.1:41310)
0806 02:58:11.776090 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:11.776194 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.188:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260806 02:58:11.776702 28295 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.211.188:46387: Network error: Client connection negotiation failed: client connection to 127.25.211.188:46387: connect: Connection refused (error 111)
I20260806 02:58:11.814926 28185 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.211.188" port: 46387 } attempt: 1
I20260806 02:58:11.816134 28188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:11.815647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:46387 (local address 127.0.0.1:41318)
0806 02:58:11.815860 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:11.815962 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.188:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260806 02:58:11.817332 28188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:11.816967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:46387 (local address 127.0.0.1:41324)
0806 02:58:11.817110 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:11.817200 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.188:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260806 02:58:11.817585 28185 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.211.188:46387: Network error: Client connection negotiation failed: client connection to 127.25.211.188:46387: connect: Connection refused (error 111)
I20260806 02:58:11.868006 28295 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.211.188" port: 46387 } attempt: 3
I20260806 02:58:11.869513 28298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:11.868813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:46387 (local address 127.0.0.1:41326)
0806 02:58:11.869169 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:11.869306 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.188:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260806 02:58:11.870980 28298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:11.870503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:46387 (local address 127.0.0.1:41328)
0806 02:58:11.870760 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:11.870848 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.188:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
W20260806 02:58:11.871225 28295 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.211.188:46387: Network error: Client connection negotiation failed: client connection to 127.25.211.188:46387: connect: Connection refused (error 111)
I20260806 02:58:11.885840 28185 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.211.188" port: 46387 } attempt: 2
I20260806 02:58:11.887149 28188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:11.886633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:46387 (local address 127.0.0.1:41330)
0806 02:58:11.886869 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:11.886961 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.188:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260806 02:58:11.888252 28188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:11.887862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:46387 (local address 127.0.0.1:41342)
0806 02:58:11.888000 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:11.888130 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.188:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260806 02:58:11.888468 28185 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.211.188:46387: Network error: Client connection negotiation failed: client connection to 127.25.211.188:46387: connect: Connection refused (error 111)
I20260806 02:58:11.994693 28185 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.211.188" port: 46387 } attempt: 3
I20260806 02:58:11.995936 28188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:11.995427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:46387 (local address 127.0.0.1:41348)
0806 02:58:11.995665 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:11.995761 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.188:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260806 02:58:11.996974 28188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:11.996608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:46387 (local address 127.0.0.1:41356)
0806 02:58:11.996804 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:11.996872 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.188:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260806 02:58:11.997187 28185 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.211.188:46387: Network error: Client connection negotiation failed: client connection to 127.25.211.188:46387: connect: Connection refused (error 111)
I20260806 02:58:12.014470 28295 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.211.188" port: 46387 } attempt: 4
I20260806 02:58:12.016376 28298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:12.015479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:46387 (local address 127.0.0.1:41362)
0806 02:58:12.015854 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:12.016033 (+ 179us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.188:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260806 02:58:12.017879 28298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:12.017430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:46387 (local address 127.0.0.1:41378)
0806 02:58:12.017626 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:12.017726 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.25.211.188:46387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260806 02:58:12.018270 28295 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.25.211.188:46387: Network error: Client connection negotiation failed: client connection to 127.25.211.188:46387: connect: Connection refused (error 111)
W20260806 02:58:12.046614 28307 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 02:58:12.046973 28307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 02:58:12.047080 28307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 02:58:12.047199 28307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 02:58:12.054821 28307 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 02:58:12.054945 28307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 02:58:12.055029 28307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 02:58:12.055096 28307 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 02:58:12.055199 28307 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 02:58:12.065493 28307 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.211.148:42775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.25.211.190:46465,127.25.211.189:33735,127.25.211.188:46387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/krb5kdc/kudu_127.25.211.188.keytab
--principal=kudu/127.25.211.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.211.188:46387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.25.211.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 01:55:54 UTC on ccycloud.quasar-csdhqu.root.comops.site
build id 4569536
ASAN enabled
I20260806 02:58:12.066954 28307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 02:58:12.068315 28307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 02:58:12.080226 28321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:58:12.080715 28320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 02:58:12.081197 28323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 02:58:12.081874 28307 server_base.cc:1034] running on GCE node
I20260806 02:58:12.082852 28307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 02:58:12.084465 28307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 02:58:12.085740 28307 hybrid_clock.cc:648] HybridClock initialized: now 1785985092085658 us; error 51 us; skew 500 ppm
I20260806 02:58:12.086193 28307 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 02:58:12 dist-test-slave-zj8q krb5kdc[28052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985092, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 02:58:12.089672 28307 init.cc:377] Logged in from keytab as kudu/127.25.211.188@KRBTEST.COM (short username kudu)
I20260806 02:58:12.091288 28307 webserver.cc:460] Webserver started at http://127.25.211.188:35803/ using document root <none> and password file <none>
I20260806 02:58:12.092007 28307 fs_manager.cc:359] Metadata directory not provided
I20260806 02:58:12.092101 28307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 02:58:12.092378 28307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 02:58:12.095498 28307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/minicluster-data/master-2/data/instance:
uuid: "d95adef0560145a6b268ac496f769d18"
format_stamp: "Formatted at 2026-08-06 02:58:12 on dist-test-slave-zj8q"
server_key: "9a69c06d1e056d9efebfd868bb8bd13a"
server_key_iv: "db48c74e50dd4efa402bc8c519cf1362"
server_key_version: "encryptionkey@0"
I20260806 02:58:12.096354 28307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/minicluster-data/master-2/wal/instance:
uuid: "d95adef0560145a6b268ac496f769d18"
format_stamp: "Formatted at 2026-08-06 02:58:12 on dist-test-slave-zj8q"
server_key: "9a69c06d1e056d9efebfd868bb8bd13a"
server_key_iv: "db48c74e50dd4efa402bc8c519cf1362"
server_key_version: "encryptionkey@0"
I20260806 02:58:12.101732 28307 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260806 02:58:12.105545 28330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 02:58:12.107122 28307 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 02:58:12.107301 28307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/minicluster-data/master-2/data,/tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/minicluster-data/master-2/wal
uuid: "d95adef0560145a6b268ac496f769d18"
format_stamp: "Formatted at 2026-08-06 02:58:12 on dist-test-slave-zj8q"
server_key: "9a69c06d1e056d9efebfd868bb8bd13a"
server_key_iv: "db48c74e50dd4efa402bc8c519cf1362"
server_key_version: "encryptionkey@0"
I20260806 02:58:12.107528 28307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 02:58:12.122260 28307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 02:58:12.126682 28307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 02:58:12.126972 28307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 02:58:12.145710 28307 rpc_server.cc:307] RPC server started. Bound to: 127.25.211.188:46387
I20260806 02:58:12.145725 28412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.25.211.188:46387 every 8 connection(s)
I20260806 02:58:12.147238 28307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/minicluster-data/master-2/data/info.pb
I20260806 02:58:12.151383 28413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 02:58:12.157547 26446 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu as pid 28307
I20260806 02:58:12.157733 26446 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFsNc4K/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1785984707115702-26446-0/minicluster-data/master-2/wal/instance
I20260806 02:58:12.158191 26446 external_mini_cluster.cc:1428] Setting key b043ea47342f47b4d495f24291a1fb10
I20260806 02:58:12.158097 28413 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.211.190" port: 46465 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 02:58:12 dist-test-slave-zj8q krb5kdc[28052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985092, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.188@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
Aug 06 02:58:12 dist-test-slave-zj8q krb5kdc[28052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.188@KRBTEST.COM
I20260806 02:58:12.174399 28185 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.211.188" port: 46387 } attempt: 4
I20260806 02:58:12.182650 28424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:12.160998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.190:46465 (local address 127.0.0.1:40110)
0806 02:58:12.161918 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:12.161958 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:12.162900 (+ 942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:12.163452 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:12.163459 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:12.164349 (+ 890us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:58:12.164966 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:12.164993 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:12.166488 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:12.166493 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:12.168264 (+ 1771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:12.168282 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:12.169413 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:12.169418 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:12.169648 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:12.170557 (+ 909us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:58:12.170592 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:58:12.173372 (+ 2780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:58:12.177863 (+ 4491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:12.177876 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:12.177891 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:12.178602 (+ 711us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:12.179059 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:12.179064 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:12.179067 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:12.179330 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:12.179997 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:58:12.180006 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:58:12.180494 (+ 488us) client_negotiation.cc:769] Sending connection context
0806 02:58:12.180813 (+ 319us) client_negotiation.cc:240] Negotiation successful
0806 02:58:12.181354 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"spinlock_wait_cycles":20736,"thread_start_us":484,"threads_started":1}
I20260806 02:58:12.182686 28419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:12.160064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41380 (local address 127.25.211.188:46387)
0806 02:58:12.160828 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:12.160846 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:12.160892 (+ 46us) server_negotiation.cc:407] Connection header received
0806 02:58:12.162888 (+ 1996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:12.162913 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:12.163198 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:12.163423 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:12.165238 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:12.166840 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:12.168619 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:12.169507 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:12.169636 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:12.174336 (+ 4700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:12.174346 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:12.174362 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:12.174395 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:12.178652 (+ 4257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:12.179524 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:12.179531 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:12.179533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:12.179655 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:12.180107 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:12.180112 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:12.180115 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:12.180602 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:12.180907 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:12.181895 (+ 988us) server_negotiation.cc:299] Negotiation successful
0806 02:58:12.182180 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":527,"spinlock_wait_cycles":17792,"thread_start_us":329,"threads_started":1}
Aug 06 02:58:12 dist-test-slave-zj8q krb5kdc[28052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985090, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.190@KRBTEST.COM for kudu/127.25.211.188@KRBTEST.COM
I20260806 02:58:12.183410 28422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:12.160988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40110 (local address 127.25.211.190:46465)
0806 02:58:12.161517 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:12.161523 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:12.162081 (+ 558us) server_negotiation.cc:407] Connection header received
0806 02:58:12.163180 (+ 1099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:12.163186 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:12.163265 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:12.163375 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:12.165182 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:12.166318 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:12.168464 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:12.169159 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:12.169312 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:12.173652 (+ 4340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:12.173661 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:12.173666 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:12.173714 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:12.177702 (+ 3988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:12.178726 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:12.178732 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:12.178736 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:12.178924 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:12.179470 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:12.179476 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:12.179479 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:12.179865 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:12.179998 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:12.182742 (+ 2744us) server_negotiation.cc:299] Negotiation successful
0806 02:58:12.182979 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":232,"threads_started":1}
I20260806 02:58:12.185055 28413 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.211.189" port: 33735 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:58:12.191780 28188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:12.175033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:46387 (local address 127.0.0.1:41384)
0806 02:58:12.175255 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:12.175275 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:12.175394 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:12.176156 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:12.176161 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:12.176652 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:58:12.176914 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:12.176928 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:12.178463 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:12.178469 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:12.179680 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:12.179692 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:12.180525 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:12.180531 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:12.180751 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:12.181352 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:58:12.181374 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:58:12.184067 (+ 2693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:58:12.189068 (+ 5001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:12.189074 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:12.189078 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:12.189531 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:12.189932 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:12.189936 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:12.189940 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:12.190175 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:12.190838 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:58:12.190843 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:58:12.191014 (+ 171us) client_negotiation.cc:769] Sending connection context
0806 02:58:12.191145 (+ 131us) client_negotiation.cc:240] Negotiation successful
0806 02:58:12.191451 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260806 02:58:12.192868 28425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:12.175221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41384 (local address 127.25.211.188:46387)
0806 02:58:12.175851 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:12.175855 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:12.175874 (+ 19us) server_negotiation.cc:407] Connection header received
0806 02:58:12.175913 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:12.175916 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:12.175974 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:12.176093 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:12.177101 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:12.178332 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:12.179794 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:12.180370 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:12.180479 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:12.184343 (+ 3864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:12.184349 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:12.184354 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:12.184391 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:12.188887 (+ 4496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:12.189658 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:12.189664 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:12.189668 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:12.189828 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:12.190313 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:12.190318 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:12.190321 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:12.190676 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:12.192080 (+ 1404us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:12.192209 (+ 129us) server_negotiation.cc:299] Negotiation successful
0806 02:58:12.192436 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":358,"threads_started":1}
Aug 06 02:58:12 dist-test-slave-zj8q krb5kdc[28052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985092, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.188@KRBTEST.COM for kudu/127.25.211.189@KRBTEST.COM
Aug 06 02:58:12 dist-test-slave-zj8q krb5kdc[28052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.190@KRBTEST.COM
I20260806 02:58:12.199167 28185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efeebd1d291141d8a6251e0b2fbd2314: Bootstrap starting.
I20260806 02:58:12.201107 28424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:12.185764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.189:33735 (local address 127.0.0.1:47774)
0806 02:58:12.185937 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:12.185956 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:12.186039 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:12.186512 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:12.186521 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:12.187091 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:58:12.187311 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:12.187325 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:12.188789 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:12.188795 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:12.190145 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:12.190193 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:12.190943 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:12.190948 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:12.191129 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:12.192329 (+ 1200us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:58:12.192345 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:58:12.195032 (+ 2687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:58:12.198642 (+ 3610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:12.198646 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:12.198649 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:12.199025 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:12.199459 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:12.199464 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:12.199467 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:12.199609 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:12.200150 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:58:12.200155 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:58:12.200306 (+ 151us) client_negotiation.cc:769] Sending connection context
0806 02:58:12.200430 (+ 124us) client_negotiation.cc:240] Negotiation successful
0806 02:58:12.200736 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260806 02:58:12.201139 28303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:12.185857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47774 (local address 127.25.211.189:33735)
0806 02:58:12.186031 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:12.186036 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:12.186067 (+ 31us) server_negotiation.cc:407] Connection header received
0806 02:58:12.186131 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:12.186136 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:12.186239 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:12.186339 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:12.187501 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:12.188609 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:12.190348 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:12.190784 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:12.190901 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:12.195266 (+ 4365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:12.195274 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:12.195279 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:12.195325 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:12.198500 (+ 3175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:12.199152 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:12.199157 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:12.199160 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:12.199353 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:12.199730 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:12.199735 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:12.199739 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:12.200024 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:12.200162 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:12.200542 (+ 380us) server_negotiation.cc:299] Negotiation successful
0806 02:58:12.200765 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 02:58:12.202466 28413 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.25.211.188" port: 46387 } has no permanent_uuid. Determining permanent_uuid...
I20260806 02:58:12.203745 28185 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P efeebd1d291141d8a6251e0b2fbd2314: Neither blocks nor log segments found. Creating new log.
I20260806 02:58:12.204545 28422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:12.186464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40124 (local address 127.25.211.190:46465)
0806 02:58:12.186919 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:12.186923 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:12.186937 (+ 14us) server_negotiation.cc:407] Connection header received
0806 02:58:12.186977 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:12.186980 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:12.187057 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:12.187154 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:12.188568 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:12.189731 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:12.191438 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:12.192005 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:12.192117 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:12.197321 (+ 5204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:12.197327 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:12.197331 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:12.197370 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:12.200838 (+ 3468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:12.201924 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:12.201930 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:12.201933 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:12.202097 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:12.202613 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:12.202617 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:12.202620 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:12.203046 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:12.203307 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:12.203910 (+ 603us) server_negotiation.cc:299] Negotiation successful
0806 02:58:12.204181 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354}
I20260806 02:58:12.205338 28185 log.cc:824] T 00000000000000000000000000000000 P efeebd1d291141d8a6251e0b2fbd2314: Log is configured to *not* fsync() on all Append() calls
I20260806 02:58:12.210027 28185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efeebd1d291141d8a6251e0b2fbd2314: No bootstrap required, opened a new log
Aug 06 02:58:12 dist-test-slave-zj8q krb5kdc[28052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985092, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.188@KRBTEST.COM for kudu/127.25.211.188@KRBTEST.COM
Aug 06 02:58:12 dist-test-slave-zj8q krb5kdc[28052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.25.211.189@KRBTEST.COM
I20260806 02:58:12.217617 28424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:12.203168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:46387 (local address 127.0.0.1:41386)
0806 02:58:12.203506 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:12.203519 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:12.203623 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:12.204128 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:12.204133 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:12.204595 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:58:12.204817 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:12.204832 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:12.206295 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:12.206300 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:12.207574 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:12.207588 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:12.208497 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:12.208504 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:12.208680 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:12.209381 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:58:12.209406 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:58:12.211846 (+ 2440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:58:12.215369 (+ 3523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:12.215375 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:12.215380 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:12.215760 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:12.216121 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:12.216126 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:12.216129 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:12.216252 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:12.216808 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:58:12.216813 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:58:12.216954 (+ 141us) client_negotiation.cc:769] Sending connection context
0806 02:58:12.217098 (+ 144us) client_negotiation.cc:240] Negotiation successful
0806 02:58:12.217353 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282}
I20260806 02:58:12.218413 28425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:12.203302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41386 (local address 127.25.211.188:46387)
0806 02:58:12.203441 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:12.203445 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:12.203587 (+ 142us) server_negotiation.cc:407] Connection header received
0806 02:58:12.203726 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:12.203731 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:12.203817 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:12.203983 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:12.204968 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:12.206112 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:12.207724 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:12.208341 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:12.209016 (+ 675us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:12.212066 (+ 3050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:12.212073 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:12.212079 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:12.212118 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:12.215223 (+ 3105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:12.215883 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:12.215887 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:12.215890 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:12.216010 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:12.216357 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:12.216360 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:12.216362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:12.216689 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:12.216821 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:12.217922 (+ 1101us) server_negotiation.cc:299] Negotiation successful
0806 02:58:12.218144 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 02:58:12.221459 28185 raft_consensus.cc:348] T 00000000000000000000000000000000 P efeebd1d291141d8a6251e0b2fbd2314 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeebd1d291141d8a6251e0b2fbd2314" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 46465 } } peers { permanent_uuid: "98f47ebe8c2441609fbdf31c49633403" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 33735 } } peers { permanent_uuid: "d95adef0560145a6b268ac496f769d18" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 46387 } }
I20260806 02:58:12.221915 28185 raft_consensus.cc:729] T 00000000000000000000000000000000 P efeebd1d291141d8a6251e0b2fbd2314 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efeebd1d291141d8a6251e0b2fbd2314, State: Initialized, Role: FOLLOWER
I20260806 02:58:12.222519 28185 consensus_queue.cc:260] T 00000000000000000000000000000000 P efeebd1d291141d8a6251e0b2fbd2314 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeebd1d291141d8a6251e0b2fbd2314" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 46465 } } peers { permanent_uuid: "98f47ebe8c2441609fbdf31c49633403" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 33735 } } peers { permanent_uuid: "d95adef0560145a6b268ac496f769d18" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 46387 } }
I20260806 02:58:12.223482 28413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d95adef0560145a6b268ac496f769d18: Bootstrap starting.
I20260806 02:58:12.223570 28429 sys_catalog.cc:455] T 00000000000000000000000000000000 P efeebd1d291141d8a6251e0b2fbd2314 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeebd1d291141d8a6251e0b2fbd2314" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 46465 } } peers { permanent_uuid: "98f47ebe8c2441609fbdf31c49633403" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 33735 } } peers { permanent_uuid: "d95adef0560145a6b268ac496f769d18" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 46387 } } }
I20260806 02:58:12.224144 28429 sys_catalog.cc:458] T 00000000000000000000000000000000 P efeebd1d291141d8a6251e0b2fbd2314 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:58:12.224850 28185 sys_catalog.cc:564] T 00000000000000000000000000000000 P efeebd1d291141d8a6251e0b2fbd2314 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:58:12.226306 28303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:12.206643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47778 (local address 127.25.211.189:33735)
0806 02:58:12.206825 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:12.206830 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:12.206848 (+ 18us) server_negotiation.cc:407] Connection header received
0806 02:58:12.206906 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:12.206911 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:12.206984 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:12.207246 (+ 262us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:12.208518 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:12.209426 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:12.211313 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:12.212020 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:12.212111 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:12.218553 (+ 6442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:12.218561 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:12.218566 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:12.218610 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:12.222543 (+ 3933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:12.223559 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:12.223565 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:12.223568 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:12.223771 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:12.224303 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:12.224308 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:12.224311 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:12.224688 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:12.224950 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:12.225641 (+ 691us) server_negotiation.cc:299] Negotiation successful
0806 02:58:12.225900 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 02:58:12.228176 28413 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d95adef0560145a6b268ac496f769d18: Neither blocks nor log segments found. Creating new log.
I20260806 02:58:12.229633 28413 log.cc:824] T 00000000000000000000000000000000 P d95adef0560145a6b268ac496f769d18: Log is configured to *not* fsync() on all Append() calls
I20260806 02:58:12.232668 26446 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 02:58:12.236096 28139 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "efeebd1d291141d8a6251e0b2fbd2314"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40124
I20260806 02:58:12.236476 28139 raft_consensus.cc:482] T 00000000000000000000000000000000 P efeebd1d291141d8a6251e0b2fbd2314 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 02:58:12.236641 28139 raft_consensus.cc:3037] T 00000000000000000000000000000000 P efeebd1d291141d8a6251e0b2fbd2314 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:58:12.238202 28413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d95adef0560145a6b268ac496f769d18: No bootstrap required, opened a new log
I20260806 02:58:12.239984 28139 raft_consensus.cc:504] T 00000000000000000000000000000000 P efeebd1d291141d8a6251e0b2fbd2314 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeebd1d291141d8a6251e0b2fbd2314" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 46465 } } peers { permanent_uuid: "98f47ebe8c2441609fbdf31c49633403" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 33735 } } peers { permanent_uuid: "d95adef0560145a6b268ac496f769d18" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 46387 } }
W20260806 02:58:12.240682 28443 catalog_manager.cc:1558] T 00000000000000000000000000000000 P efeebd1d291141d8a6251e0b2fbd2314: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 02:58:12.240818 28443 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 02:58:12.244066 28139 leader_election.cc:288] T 00000000000000000000000000000000 P efeebd1d291141d8a6251e0b2fbd2314 [CANDIDATE]: Term 1 election: Requested vote from peers 98f47ebe8c2441609fbdf31c49633403 (127.25.211.189:33735), d95adef0560145a6b268ac496f769d18 (127.25.211.188:46387)
I20260806 02:58:12.249620 28413 raft_consensus.cc:348] T 00000000000000000000000000000000 P d95adef0560145a6b268ac496f769d18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeebd1d291141d8a6251e0b2fbd2314" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 46465 } } peers { permanent_uuid: "98f47ebe8c2441609fbdf31c49633403" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 33735 } } peers { permanent_uuid: "d95adef0560145a6b268ac496f769d18" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 46387 } }
I20260806 02:58:12.250094 28413 raft_consensus.cc:729] T 00000000000000000000000000000000 P d95adef0560145a6b268ac496f769d18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d95adef0560145a6b268ac496f769d18, State: Initialized, Role: FOLLOWER
I20260806 02:58:12.250694 28413 consensus_queue.cc:260] T 00000000000000000000000000000000 P d95adef0560145a6b268ac496f769d18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeebd1d291141d8a6251e0b2fbd2314" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 46465 } } peers { permanent_uuid: "98f47ebe8c2441609fbdf31c49633403" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 33735 } } peers { permanent_uuid: "d95adef0560145a6b268ac496f769d18" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 46387 } }
I20260806 02:58:12.247480 28071 master_runner.cc:186] Error getting master registration for 127.25.211.189:33735: OK, instance_id { permanent_uuid: "98f47ebe8c2441609fbdf31c49633403" instance_seqno: 1785985091582467 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 02:58:12.259361 28071 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 02:58:12.268473 28446 sys_catalog.cc:455] T 00000000000000000000000000000000 P d95adef0560145a6b268ac496f769d18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeebd1d291141d8a6251e0b2fbd2314" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 46465 } } peers { permanent_uuid: "98f47ebe8c2441609fbdf31c49633403" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 33735 } } peers { permanent_uuid: "d95adef0560145a6b268ac496f769d18" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 46387 } } }
I20260806 02:58:12.268999 28446 sys_catalog.cc:458] T 00000000000000000000000000000000 P d95adef0560145a6b268ac496f769d18 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:58:12.270143 28413 sys_catalog.cc:564] T 00000000000000000000000000000000 P d95adef0560145a6b268ac496f769d18 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:58:12.270325 28373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "efeebd1d291141d8a6251e0b2fbd2314" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d95adef0560145a6b268ac496f769d18"
W20260806 02:58:12.270792 28103 tablet.cc:2369] T 00000000000000000000000000000000 P efeebd1d291141d8a6251e0b2fbd2314: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:58:12.270866 28373 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d95adef0560145a6b268ac496f769d18 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:58:12.282282 28373 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d95adef0560145a6b268ac496f769d18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efeebd1d291141d8a6251e0b2fbd2314 in term 1.
I20260806 02:58:12.285676 28095 leader_election.cc:302] T 00000000000000000000000000000000 P efeebd1d291141d8a6251e0b2fbd2314 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d95adef0560145a6b268ac496f769d18, efeebd1d291141d8a6251e0b2fbd2314; no voters:
I20260806 02:58:12.286476 28429 raft_consensus.cc:2781] T 00000000000000000000000000000000 P efeebd1d291141d8a6251e0b2fbd2314 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 02:58:12.300505 28422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:12.269438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40140 (local address 127.25.211.190:46465)
0806 02:58:12.269982 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:12.269988 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:12.276974 (+ 6986us) server_negotiation.cc:407] Connection header received
0806 02:58:12.277587 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:12.277594 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:12.277808 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:12.277919 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:12.280510 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:12.281754 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:12.283983 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:12.284682 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:12.284794 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:12.292012 (+ 7218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:12.292020 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:12.292025 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:12.292121 (+ 96us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:12.296445 (+ 4324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:12.297646 (+ 1201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:12.297653 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:12.297657 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:12.297835 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:12.298397 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:12.298403 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:12.298406 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:12.298865 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:12.299164 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:12.299795 (+ 631us) server_negotiation.cc:299] Negotiation successful
0806 02:58:12.300069 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380}
I20260806 02:58:12.317492 28295 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.25.211.188" port: 46387 } attempt: 5
I20260806 02:58:12.321408 28425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:12.269642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41398 (local address 127.25.211.188:46387)
0806 02:58:12.270122 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:12.270132 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:12.301659 (+ 31527us) server_negotiation.cc:407] Connection header received
0806 02:58:12.301984 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:12.301991 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:12.302083 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:12.302195 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:12.304131 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:12.305306 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:12.307311 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:12.308041 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:12.308169 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:12.312764 (+ 4595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:12.312772 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:12.312778 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:12.312835 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:12.316916 (+ 4081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:12.318010 (+ 1094us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:12.318017 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:12.318020 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:12.318296 (+ 276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:12.318971 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:12.318978 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:12.318981 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:12.319440 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:12.319985 (+ 545us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:12.320719 (+ 734us) server_negotiation.cc:299] Negotiation successful
0806 02:58:12.320989 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370}
I20260806 02:58:12.322326 28429 raft_consensus.cc:686] T 00000000000000000000000000000000 P efeebd1d291141d8a6251e0b2fbd2314 [term 1 LEADER]: Becoming Leader. State: Replica: efeebd1d291141d8a6251e0b2fbd2314, State: Running, Role: LEADER
I20260806 02:58:12.323134 28429 consensus_queue.cc:237] T 00000000000000000000000000000000 P efeebd1d291141d8a6251e0b2fbd2314 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeebd1d291141d8a6251e0b2fbd2314" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 46465 } } peers { permanent_uuid: "98f47ebe8c2441609fbdf31c49633403" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 33735 } } peers { permanent_uuid: "d95adef0560145a6b268ac496f769d18" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 46387 } }
Aug 06 02:58:12 dist-test-slave-zj8q krb5kdc[28052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1785985091, etypes {rep=17 tkt=17 ses=17}, kudu/127.25.211.189@KRBTEST.COM for kudu/127.25.211.188@KRBTEST.COM
I20260806 02:58:12.336205 28298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:12.318317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.25.211.188:46387 (local address 127.0.0.1:41404)
0806 02:58:12.318843 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0806 02:58:12.318865 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 02:58:12.318986 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 02:58:12.320137 (+ 1151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 02:58:12.320142 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 02:58:12.320705 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0806 02:58:12.320927 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:12.320940 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:12.322390 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:12.322396 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:12.323358 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 02:58:12.323373 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 02:58:12.324677 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 02:58:12.324683 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 02:58:12.324933 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:12.325659 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 02:58:12.325684 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 02:58:12.328728 (+ 3044us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 02:58:12.333001 (+ 4273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:12.333007 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:12.333012 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:12.333566 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:12.334043 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 02:58:12.334048 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 02:58:12.334052 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 02:58:12.334293 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 02:58:12.335038 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 02:58:12.335044 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 02:58:12.335223 (+ 179us) client_negotiation.cc:769] Sending connection context
0806 02:58:12.335381 (+ 158us) client_negotiation.cc:240] Negotiation successful
0806 02:58:12.335709 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375}
I20260806 02:58:12.337378 28419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:12.319234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41404 (local address 127.25.211.188:46387)
0806 02:58:12.319441 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:12.319447 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:12.319468 (+ 21us) server_negotiation.cc:407] Connection header received
0806 02:58:12.319517 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:12.319522 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:12.319799 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:12.319926 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:12.321103 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:12.322211 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:12.323637 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:12.324315 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:12.324459 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:12.328985 (+ 4526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:12.328993 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:12.328998 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:12.329038 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:12.332806 (+ 3768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:12.333725 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:12.333731 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:12.333735 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:12.333905 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:12.334431 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:12.334437 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:12.334441 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:12.334810 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:12.334958 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:12.336786 (+ 1828us) server_negotiation.cc:299] Negotiation successful
0806 02:58:12.337043 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 02:58:12.344179 28295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98f47ebe8c2441609fbdf31c49633403: Bootstrap starting.
I20260806 02:58:12.348676 28295 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98f47ebe8c2441609fbdf31c49633403: Neither blocks nor log segments found. Creating new log.
I20260806 02:58:12.350121 28295 log.cc:824] T 00000000000000000000000000000000 P 98f47ebe8c2441609fbdf31c49633403: Log is configured to *not* fsync() on all Append() calls
I20260806 02:58:12.351158 28462 sys_catalog.cc:455] T 00000000000000000000000000000000 P efeebd1d291141d8a6251e0b2fbd2314 [sys.catalog]: SysCatalogTable state changed. Reason: New leader efeebd1d291141d8a6251e0b2fbd2314. Latest consensus state: current_term: 1 leader_uuid: "efeebd1d291141d8a6251e0b2fbd2314" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeebd1d291141d8a6251e0b2fbd2314" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 46465 } } peers { permanent_uuid: "98f47ebe8c2441609fbdf31c49633403" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 33735 } } peers { permanent_uuid: "d95adef0560145a6b268ac496f769d18" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 46387 } } }
I20260806 02:58:12.351948 28462 sys_catalog.cc:458] T 00000000000000000000000000000000 P efeebd1d291141d8a6251e0b2fbd2314 [sys.catalog]: This master's current role is: LEADER
I20260806 02:58:12.360925 28295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98f47ebe8c2441609fbdf31c49633403: No bootstrap required, opened a new log
I20260806 02:58:12.363547 28472 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 02:58:12.366407 28472 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 02:58:12.372942 28295 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98f47ebe8c2441609fbdf31c49633403 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeebd1d291141d8a6251e0b2fbd2314" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 46465 } } peers { permanent_uuid: "98f47ebe8c2441609fbdf31c49633403" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 33735 } } peers { permanent_uuid: "d95adef0560145a6b268ac496f769d18" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 46387 } }
I20260806 02:58:12.373960 28295 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98f47ebe8c2441609fbdf31c49633403 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f47ebe8c2441609fbdf31c49633403, State: Initialized, Role: FOLLOWER
I20260806 02:58:12.374943 28295 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98f47ebe8c2441609fbdf31c49633403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeebd1d291141d8a6251e0b2fbd2314" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 46465 } } peers { permanent_uuid: "98f47ebe8c2441609fbdf31c49633403" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 33735 } } peers { permanent_uuid: "d95adef0560145a6b268ac496f769d18" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 46387 } }
I20260806 02:58:12.374388 28303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:12.269530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47782 (local address 127.25.211.189:33735)
0806 02:58:12.269737 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:12.269744 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:12.316137 (+ 46393us) server_negotiation.cc:407] Connection header received
0806 02:58:12.316665 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:12.316672 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:12.316773 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:12.316860 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:12.334605 (+ 17745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:12.335904 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:12.341130 (+ 5226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:12.341906 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:12.342020 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:12.363262 (+ 21242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 02:58:12.363271 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 02:58:12.363281 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 02:58:12.363336 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0806 02:58:12.368072 (+ 4736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:12.369440 (+ 1368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:12.369446 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:12.369450 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:12.369817 (+ 367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 02:58:12.370841 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 02:58:12.370848 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 02:58:12.370852 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 02:58:12.371605 (+ 753us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 02:58:12.372087 (+ 482us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:12.373302 (+ 1215us) server_negotiation.cc:299] Negotiation successful
0806 02:58:12.373742 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260806 02:58:12.379177 28338 tablet.cc:2369] T 00000000000000000000000000000000 P d95adef0560145a6b268ac496f769d18: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 02:58:12.379411 28295 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98f47ebe8c2441609fbdf31c49633403 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 02:58:12.380474 28253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "efeebd1d291141d8a6251e0b2fbd2314" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "98f47ebe8c2441609fbdf31c49633403"
I20260806 02:58:12.381072 28253 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98f47ebe8c2441609fbdf31c49633403 [term 0 FOLLOWER]: Advancing to term 1
I20260806 02:58:12.389722 28253 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 98f47ebe8c2441609fbdf31c49633403 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efeebd1d291141d8a6251e0b2fbd2314 in term 1.
I20260806 02:58:12.378190 28479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98f47ebe8c2441609fbdf31c49633403 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeebd1d291141d8a6251e0b2fbd2314" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 46465 } } peers { permanent_uuid: "98f47ebe8c2441609fbdf31c49633403" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 33735 } } peers { permanent_uuid: "d95adef0560145a6b268ac496f769d18" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 46387 } } }
I20260806 02:58:12.391156 28479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98f47ebe8c2441609fbdf31c49633403 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:58:12.407933 28373 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d95adef0560145a6b268ac496f769d18 [term 1 FOLLOWER]: Refusing update from remote peer efeebd1d291141d8a6251e0b2fbd2314: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:58:12.409220 28462 consensus_queue.cc:1035] T 00000000000000000000000000000000 P efeebd1d291141d8a6251e0b2fbd2314 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d95adef0560145a6b268ac496f769d18" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 46387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:58:12.412114 28488 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 02:58:12.416002 28253 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 98f47ebe8c2441609fbdf31c49633403 [term 1 FOLLOWER]: Refusing update from remote peer efeebd1d291141d8a6251e0b2fbd2314: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 02:58:12.430394 28462 consensus_queue.cc:1035] T 00000000000000000000000000000000 P efeebd1d291141d8a6251e0b2fbd2314 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98f47ebe8c2441609fbdf31c49633403" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 33735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 02:58:12.424438 28307 master_runner.cc:186] Error getting master registration for 127.25.211.189:33735: OK, instance_id { permanent_uuid: "98f47ebe8c2441609fbdf31c49633403" instance_seqno: 1785985091582467 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 02:58:12.451490 28307 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 02:58:12.459049 28479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98f47ebe8c2441609fbdf31c49633403 [sys.catalog]: SysCatalogTable state changed. Reason: New leader efeebd1d291141d8a6251e0b2fbd2314. Latest consensus state: current_term: 1 leader_uuid: "efeebd1d291141d8a6251e0b2fbd2314" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeebd1d291141d8a6251e0b2fbd2314" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 46465 } } peers { permanent_uuid: "98f47ebe8c2441609fbdf31c49633403" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 33735 } } peers { permanent_uuid: "d95adef0560145a6b268ac496f769d18" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 46387 } } }
I20260806 02:58:12.459455 28479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98f47ebe8c2441609fbdf31c49633403 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:58:12.462107 28482 mvcc.cc:205] Tried to move back new op lower bound from 7315394938465550336 to 7315394938164822016. Current Snapshot: MvccSnapshot[applied={T|T < 7315394938465550336}]
I20260806 02:58:12.465490 28479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98f47ebe8c2441609fbdf31c49633403 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "efeebd1d291141d8a6251e0b2fbd2314" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeebd1d291141d8a6251e0b2fbd2314" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 46465 } } peers { permanent_uuid: "98f47ebe8c2441609fbdf31c49633403" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 33735 } } peers { permanent_uuid: "d95adef0560145a6b268ac496f769d18" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 46387 } } }
I20260806 02:58:12.465830 28479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98f47ebe8c2441609fbdf31c49633403 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:58:12.472497 28429 sys_catalog.cc:455] T 00000000000000000000000000000000 P efeebd1d291141d8a6251e0b2fbd2314 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "efeebd1d291141d8a6251e0b2fbd2314" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeebd1d291141d8a6251e0b2fbd2314" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 46465 } } peers { permanent_uuid: "98f47ebe8c2441609fbdf31c49633403" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 33735 } } peers { permanent_uuid: "d95adef0560145a6b268ac496f769d18" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 46387 } } }
I20260806 02:58:12.472978 28429 sys_catalog.cc:458] T 00000000000000000000000000000000 P efeebd1d291141d8a6251e0b2fbd2314 [sys.catalog]: This master's current role is: LEADER
I20260806 02:58:12.473340 28498 mvcc.cc:205] Tried to move back new op lower bound from 7315394938465550336 to 7315394938164822016. Current Snapshot: MvccSnapshot[applied={T|T < 7315394938465550336 or (T in {7315394938465550336})}]
I20260806 02:58:12.475920 28472 catalog_manager.cc:1347] Generated new cluster ID: 9e82b65220864f8db2076eb3d05c27cc
I20260806 02:58:12.476052 28472 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 02:58:12.476477 28511 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 02:58:12.480643 28446 sys_catalog.cc:455] T 00000000000000000000000000000000 P d95adef0560145a6b268ac496f769d18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader efeebd1d291141d8a6251e0b2fbd2314. Latest consensus state: current_term: 1 leader_uuid: "efeebd1d291141d8a6251e0b2fbd2314" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeebd1d291141d8a6251e0b2fbd2314" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 46465 } } peers { permanent_uuid: "98f47ebe8c2441609fbdf31c49633403" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 33735 } } peers { permanent_uuid: "d95adef0560145a6b268ac496f769d18" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 46387 } } }
I20260806 02:58:12.481011 28446 sys_catalog.cc:458] T 00000000000000000000000000000000 P d95adef0560145a6b268ac496f769d18 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:58:12.486841 28429 sys_catalog.cc:455] T 00000000000000000000000000000000 P efeebd1d291141d8a6251e0b2fbd2314 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "efeebd1d291141d8a6251e0b2fbd2314" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeebd1d291141d8a6251e0b2fbd2314" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 46465 } } peers { permanent_uuid: "98f47ebe8c2441609fbdf31c49633403" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 33735 } } peers { permanent_uuid: "d95adef0560145a6b268ac496f769d18" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 46387 } } }
I20260806 02:58:12.487174 28429 sys_catalog.cc:458] T 00000000000000000000000000000000 P efeebd1d291141d8a6251e0b2fbd2314 [sys.catalog]: This master's current role is: LEADER
I20260806 02:58:12.490592 28446 sys_catalog.cc:455] T 00000000000000000000000000000000 P d95adef0560145a6b268ac496f769d18 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "efeebd1d291141d8a6251e0b2fbd2314" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efeebd1d291141d8a6251e0b2fbd2314" member_type: VOTER last_known_addr { host: "127.25.211.190" port: 46465 } } peers { permanent_uuid: "98f47ebe8c2441609fbdf31c49633403" member_type: VOTER last_known_addr { host: "127.25.211.189" port: 33735 } } peers { permanent_uuid: "d95adef0560145a6b268ac496f769d18" member_type: VOTER last_known_addr { host: "127.25.211.188" port: 46387 } } }
I20260806 02:58:12.490875 28446 sys_catalog.cc:458] T 00000000000000000000000000000000 P d95adef0560145a6b268ac496f769d18 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 02:58:12.520628 28472 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 02:58:12.522948 28472 catalog_manager.cc:1504] Loading token signing keys...
I20260806 02:58:12.535229 28472 catalog_manager.cc:5868] T 00000000000000000000000000000000 P efeebd1d291141d8a6251e0b2fbd2314: Generated new TSK 0
I20260806 02:58:12.536475 28472 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 02:58:12.631456 28303 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:58:12.620838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47794 (local address 127.25.211.189:33735)
0806 02:58:12.621075 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:12.621082 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:12.629358 (+ 8276us) server_negotiation.cc:407] Connection header received
0806 02:58:12.629582 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:12.629589 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:12.629688 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:12.629837 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:12.631217 (+ 1380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47794: BlockingRecv error: recv got EOF from 127.0.0.1:47794 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260806 02:58:12.631582 28422 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:58:12.621245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40148 (local address 127.25.211.190:46465)
0806 02:58:12.621468 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:12.621474 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:12.628986 (+ 7512us) server_negotiation.cc:407] Connection header received
0806 02:58:12.629719 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:12.629726 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:12.629822 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:12.629896 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:12.631276 (+ 1380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40148: BlockingRecv error: recv got EOF from 127.0.0.1:40148 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260806 02:58:12.633775 28419 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:58:12.620942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41418 (local address 127.25.211.188:46387)
0806 02:58:12.621142 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:12.621148 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:12.632141 (+ 10993us) server_negotiation.cc:407] Connection header received
0806 02:58:12.632413 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:12.632419 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:12.632502 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:12.632579 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:12.633539 (+ 960us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41418: BlockingRecv error: recv got EOF from 127.0.0.1:41418 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260806 02:58:12.639330 28303 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:58:12.637421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47800 (local address 127.25.211.189:33735)
0806 02:58:12.637609 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:12.637616 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:12.637882 (+ 266us) server_negotiation.cc:407] Connection header received
0806 02:58:12.638077 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:12.638083 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:12.638246 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:12.638359 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:12.639044 (+ 685us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47800: BlockingRecv error: recv got EOF from 127.0.0.1:47800 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260806 02:58:12.645560 28303 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:58:12.643899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47814 (local address 127.25.211.189:33735)
0806 02:58:12.644090 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:12.644097 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:12.644216 (+ 119us) server_negotiation.cc:407] Connection header received
0806 02:58:12.644379 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:12.644385 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:12.644474 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:12.644586 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:12.645321 (+ 735us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47814: BlockingRecv error: recv got EOF from 127.0.0.1:47814 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260806 02:58:12.652707 28303 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:58:12.650512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47822 (local address 127.25.211.189:33735)
0806 02:58:12.650736 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:12.650742 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:12.651108 (+ 366us) server_negotiation.cc:407] Connection header received
0806 02:58:12.651303 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:12.651306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:12.651393 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:12.651478 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:12.652466 (+ 988us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47822: BlockingRecv error: recv got EOF from 127.0.0.1:47822 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260806 02:58:12.662523 28303 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:58:12.660523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47832 (local address 127.25.211.189:33735)
0806 02:58:12.660759 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:12.660765 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:12.661033 (+ 268us) server_negotiation.cc:407] Connection header received
0806 02:58:12.661191 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:12.661197 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:12.661293 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:12.661415 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:12.662276 (+ 861us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47832: BlockingRecv error: recv got EOF from 127.0.0.1:47832 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
W20260806 02:58:12.676676 28303 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:58:12.674295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47844 (local address 127.25.211.189:33735)
0806 02:58:12.674539 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:12.674546 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:12.675025 (+ 479us) server_negotiation.cc:407] Connection header received
0806 02:58:12.675301 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:12.675305 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:12.675384 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:12.675479 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:12.676401 (+ 922us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47844: BlockingRecv error: recv got EOF from 127.0.0.1:47844 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260806 02:58:12.699172 28303 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:58:12.697112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47858 (local address 127.25.211.189:33735)
0806 02:58:12.697326 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:12.697333 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:12.697644 (+ 311us) server_negotiation.cc:407] Connection header received
0806 02:58:12.697825 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:12.697831 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:12.697936 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:12.698059 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:12.698914 (+ 855us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47858: BlockingRecv error: recv got EOF from 127.0.0.1:47858 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260806 02:58:12.737428 28303 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:58:12.735164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50328 (local address 127.25.211.189:33735)
0806 02:58:12.735365 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:12.735372 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:12.735719 (+ 347us) server_negotiation.cc:407] Connection header received
0806 02:58:12.735924 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:12.735931 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:12.736053 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:12.736217 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:12.737153 (+ 936us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50328: BlockingRecv error: recv got EOF from 127.0.0.1:50328 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260806 02:58:12.806841 28303 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:58:12.805074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50342 (local address 127.25.211.189:33735)
0806 02:58:12.805299 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:12.805304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:12.805570 (+ 266us) server_negotiation.cc:407] Connection header received
0806 02:58:12.805754 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:12.805758 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:12.805835 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:12.805931 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:12.806669 (+ 738us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50342: BlockingRecv error: recv got EOF from 127.0.0.1:50342 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
W20260806 02:58:12.949961 28303 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:58:12.947912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50346 (local address 127.25.211.189:33735)
0806 02:58:12.948160 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:12.948167 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:12.948434 (+ 267us) server_negotiation.cc:407] Connection header received
0806 02:58:12.948657 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:12.948662 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:12.948738 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:12.948842 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:12.949688 (+ 846us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50346: BlockingRecv error: recv got EOF from 127.0.0.1:50346 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
W20260806 02:58:13.211810 28303 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 02:58:13.209808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50354 (local address 127.25.211.189:33735)
0806 02:58:13.210046 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:13.210053 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:13.210282 (+ 229us) server_negotiation.cc:407] Connection header received
0806 02:58:13.210504 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:13.210508 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:13.210584 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:13.210674 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 02:58:13.211578 (+ 904us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50354: BlockingRecv error: recv got EOF from 127.0.0.1:50354 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
I20260806 02:58:13.416800 28488 catalog_manager.cc:1259] Loaded cluster ID: 9e82b65220864f8db2076eb3d05c27cc
I20260806 02:58:13.416911 28488 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d95adef0560145a6b268ac496f769d18: loading cluster ID for follower catalog manager: success
I20260806 02:58:13.419775 28488 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d95adef0560145a6b268ac496f769d18: acquiring CA information for follower catalog manager: success
I20260806 02:58:13.421830 28488 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d95adef0560145a6b268ac496f769d18: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 02:58:13.482323 28511 catalog_manager.cc:1259] Loaded cluster ID: 9e82b65220864f8db2076eb3d05c27cc
I20260806 02:58:13.482457 28511 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 98f47ebe8c2441609fbdf31c49633403: loading cluster ID for follower catalog manager: success
I20260806 02:58:13.486634 28511 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 98f47ebe8c2441609fbdf31c49633403: acquiring CA information for follower catalog manager: success
I20260806 02:58:13.489395 28511 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 98f47ebe8c2441609fbdf31c49633403: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 02:58:13.739645 28627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:13.728377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50360 (local address 127.25.211.189:33735)
0806 02:58:13.730596 (+ 2219us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:13.730604 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:13.731015 (+ 411us) server_negotiation.cc:407] Connection header received
0806 02:58:13.731303 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:13.731310 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:13.731541 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:13.731688 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:58:13.732574 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:13.733992 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:13.736328 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:13.737049 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:13.737204 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:13.737947 (+ 743us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:58:13.738840 (+ 893us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:58:13.739005 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:13.739135 (+ 130us) server_negotiation.cc:299] Negotiation successful
0806 02:58:13.739240 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2070,"thread_start_us":1910,"threads_started":1}
I20260806 02:58:13.774690 28637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:13.752502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47724 (local address 127.25.211.188:46387)
0806 02:58:13.765568 (+ 13066us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:13.765575 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:13.766461 (+ 886us) server_negotiation.cc:407] Connection header received
0806 02:58:13.766724 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:13.766730 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:13.766955 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:13.767166 (+ 211us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:58:13.767757 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:13.769020 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:13.771078 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:13.771830 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:13.771968 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:13.772679 (+ 711us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:58:13.773519 (+ 840us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:58:13.773660 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:13.774173 (+ 513us) server_negotiation.cc:299] Negotiation successful
0806 02:58:13.774273 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12854,"thread_start_us":13089,"threads_started":1}
I20260806 02:58:13.798790 28637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:13.787730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47732 (local address 127.25.211.188:46387)
0806 02:58:13.789949 (+ 2219us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:13.789956 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:13.791352 (+ 1396us) server_negotiation.cc:407] Connection header received
0806 02:58:13.791690 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:13.791697 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:13.791793 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:13.791862 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:58:13.792692 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:13.793822 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:13.796278 (+ 2456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:13.796908 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:13.797013 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:13.797813 (+ 800us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:58:13.798007 (+ 194us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:58:13.798081 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:13.798357 (+ 276us) server_negotiation.cc:299] Negotiation successful
0806 02:58:13.798453 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2078}
I20260806 02:58:13.799757 28645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:13.786662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60914 (local address 127.25.211.190:46465)
0806 02:58:13.789949 (+ 3287us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:13.789956 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:13.791352 (+ 1396us) server_negotiation.cc:407] Connection header received
0806 02:58:13.791927 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:13.791934 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:13.792178 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:13.792345 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:58:13.793120 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:13.794168 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:13.797114 (+ 2946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:13.797746 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:13.797842 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:13.798625 (+ 783us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:58:13.799116 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:58:13.799196 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:13.799411 (+ 215us) server_negotiation.cc:299] Negotiation successful
0806 02:58:13.799495 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3168,"thread_start_us":3053,"threads_started":1}
I20260806 02:58:13.803849 28627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 02:58:13.790456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50362 (local address 127.25.211.189:33735)
0806 02:58:13.790650 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0806 02:58:13.790657 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 02:58:13.796568 (+ 5911us) server_negotiation.cc:407] Connection header received
0806 02:58:13.796751 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 02:58:13.796758 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 02:58:13.796872 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 02:58:13.796988 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 02:58:13.798026 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:13.799160 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:13.801892 (+ 2732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 02:58:13.802437 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 02:58:13.802539 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 02:58:13.803110 (+ 571us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 02:58:13.803221 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 02:58:13.803290 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 02:58:13.803474 (+ 184us) server_negotiation.cc:299] Negotiation successful
0806 02:58:13.803543 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 02:58:13.809577 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 28071
I20260806 02:58:13.810895 28174 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:58:13.924103 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 28071
W20260806 02:58:13.951397 28215 connection.cc:535] server connection from 127.0.0.1:47764 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 02:58:13.951915 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 28193
I20260806 02:58:13.953222 28289 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:58:14.065317 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 28193
I20260806 02:58:14.090585 26446 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu pid 28307
I20260806 02:58:14.091701 28408 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 02:58:14.201354 26446 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFsNc4K/build/asan/bin/kudu with pid 28307
2026-08-06T02:58:14Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5266 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (11058 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (387041 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST